Nov 02 19:25:13.406011 ip-10-0-131-244 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 19:25:13.406022 ip-10-0-131-244 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 19:25:13.406029 ip-10-0-131-244 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 19:25:13.406247 ip-10-0-131-244 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 19:25:23.466114 ip-10-0-131-244 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 19:25:23.466139 ip-10-0-131-244 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 134be83f6f7c42ff9df4f9a419ff198f -- Nov 02 19:27:47.037775 ip-10-0-131-244 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:27:47.464101 ip-10-0-131-244 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:27:47.464101 ip-10-0-131-244 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:27:47.464101 ip-10-0-131-244 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:27:47.464101 ip-10-0-131-244 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:27:47.468121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468033 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:47.468121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468120 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:47.468121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468124 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468127 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468130 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468133 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468136 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468139 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468142 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468145 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468147 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468150 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468153 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468156 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468158 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468161 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468163 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468166 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468169 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468172 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468174 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468185 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:47.468220 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468188 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468191 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468193 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468196 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468199 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468202 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468205 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468208 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468210 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468213 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468216 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468219 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468222 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468224 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468227 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468230 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468233 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468235 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468238 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468241 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:47.468690 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468244 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468246 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468249 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468252 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468254 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468257 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468260 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468262 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468265 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468267 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468270 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468272 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468275 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468278 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468280 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468283 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468286 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468289 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468293 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468296 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:47.469171 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468298 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468301 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468304 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468306 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468309 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468311 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468314 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468316 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468319 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468322 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468325 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468327 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468330 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468333 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468337 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468341 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468344 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468347 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468350 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:47.469654 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468353 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468355 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468358 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468361 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468364 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468366 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468369 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468371 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468374 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468377 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468380 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468382 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468385 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468388 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468390 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468394 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468396 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468399 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468404 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:47.470111 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468408 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468411 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468414 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468416 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468419 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.468422 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469217 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469224 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469227 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469230 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469234 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469237 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469239 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469242 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469245 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469247 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469250 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469252 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469255 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469258 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:47.470617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469261 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469263 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469266 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469269 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469273 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469275 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469278 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469281 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469283 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469286 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469290 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469294 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469297 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469300 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469302 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469305 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469307 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469310 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469314 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469316 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:47.471093 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469319 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469321 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469324 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469327 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469330 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469333 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469335 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469339 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469343 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469347 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469350 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469353 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469355 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469358 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469361 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469364 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469367 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469369 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469372 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:47.471581 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469375 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469378 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469381 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469384 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469387 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469389 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469392 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469394 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469397 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469399 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469402 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469405 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469408 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469410 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469413 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469415 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469418 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469421 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469423 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:47.472032 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469426 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469429 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469431 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469434 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469437 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469439 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469442 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469444 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469447 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469449 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469453 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469456 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469458 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469461 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469464 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469467 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469484 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469487 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469490 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:47.472492 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469492 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469495 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469497 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469500 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469503 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469505 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469508 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469511 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469514 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469516 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469519 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469522 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469525 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469527 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.469530 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470032 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470037 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470040 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470043 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470046 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:47.472945 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470049 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470051 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470054 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470057 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470059 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470062 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470065 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470067 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470070 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470072 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470075 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470077 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470080 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470083 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470086 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470088 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470091 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470094 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470096 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470100 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:47.473423 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470102 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470105 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470107 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470110 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470112 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470115 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470118 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470121 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470124 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470126 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470130 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470132 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470134 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470137 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470139 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470142 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470145 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470147 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470151 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:47.473923 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470155 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470158 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470161 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470164 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470167 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470169 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470172 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470174 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470177 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470179 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470182 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470184 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470187 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470190 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470192 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470195 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470197 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470200 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470203 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:47.474367 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470206 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470209 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470213 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470216 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470219 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470222 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470225 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470228 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470231 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470233 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470236 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470239 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470241 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470244 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470247 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470249 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470252 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470254 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470257 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:47.474825 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470260 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470263 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470266 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470268 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470271 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470273 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470275 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470279 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470282 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470284 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470287 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470289 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470292 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470294 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470297 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470299 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470302 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470305 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470308 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:47.475303 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470310 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470312 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470315 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470317 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.470320 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471385 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471400 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471408 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471413 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471418 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471422 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471426 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471431 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471434 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471437 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471442 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471445 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471449 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471452 2573 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471455 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471460 2573 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471463 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471468 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471482 2573 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:27:47.475769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471486 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471489 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471492 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471496 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471500 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471504 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471507 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471510 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471514 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471517 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471520 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471524 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471527 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471531 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471535 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471538 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471541 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471545 2573 flags.go:64] FLAG: --enable-server="true" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471548 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471552 2573 flags.go:64] FLAG: --event-burst="100" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471555 2573 flags.go:64] FLAG: --event-qps="50" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471559 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471562 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471565 2573 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:27:47.476434 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471570 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471573 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471576 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471580 2573 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471584 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471588 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471591 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471595 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471598 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471601 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471604 2573 flags.go:64] FLAG: --feature-gates="" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471609 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471612 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471615 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471619 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471622 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471625 2573 flags.go:64] FLAG: --help="false" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471628 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471631 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471635 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471638 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471641 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471645 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:27:47.477065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471648 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471652 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471655 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471658 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471661 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471665 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471667 2573 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471671 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471673 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471677 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471680 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471683 2573 flags.go:64] FLAG: --lock-file="" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471686 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471691 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471694 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471705 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471708 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471714 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471718 2573 flags.go:64] FLAG: --logging-format="text" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471720 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471724 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471727 2573 flags.go:64] FLAG: --manifest-url="" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471730 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471735 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471738 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:27:47.477668 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471742 2573 flags.go:64] FLAG: --max-pods="110" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471745 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471748 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471752 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471755 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471758 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471761 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471766 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471774 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471777 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471781 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471784 2573 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471787 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471791 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471794 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471797 2573 flags.go:64] FLAG: --port="10250" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471800 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471803 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e490ebd6853ea3b9" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471806 2573 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471809 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471813 2573 flags.go:64] FLAG: --register-node="true" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471817 2573 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471821 2573 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471824 2573 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471827 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471830 2573 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:27:47.478589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471834 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471838 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471841 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471844 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471847 2573 flags.go:64] FLAG: --runonce="false" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471850 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471853 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471856 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471859 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471862 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471866 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471869 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471872 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471877 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471880 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471883 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471886 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471890 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471893 2573 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471896 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471901 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471905 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471907 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471912 2573 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471915 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:27:47.479530 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471918 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471922 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471925 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471928 2573 flags.go:64] FLAG: --v="2" Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471933 2573 flags.go:64] FLAG: --version="false" Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471937 2573 flags.go:64] FLAG: --vmodule="" Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471941 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.471945 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472088 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472092 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472095 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472100 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472103 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472106 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472109 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472112 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472115 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472118 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472120 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472123 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472127 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:47.480497 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472130 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472132 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472135 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472138 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472140 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472143 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472146 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472149 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472151 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472154 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472157 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472159 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472163 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472166 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472169 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472172 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472174 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472177 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472180 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:47.481121 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472184 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472187 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472189 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472192 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472195 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472198 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472200 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472203 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472206 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472209 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472211 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472214 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472218 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472220 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472223 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472225 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472228 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472231 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472233 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:47.481828 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472236 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472239 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472241 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472244 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472246 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472249 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472253 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472256 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472258 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472261 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472264 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472266 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472269 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472272 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472275 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472278 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472281 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472284 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472287 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472289 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:47.482771 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472292 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472295 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472297 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472300 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472302 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472307 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472310 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472313 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472315 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472318 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472321 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472323 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472326 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472329 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472331 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472334 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472336 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472339 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472343 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472345 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:47.483656 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472348 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472351 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472353 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472356 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472359 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472361 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472364 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472367 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472370 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472373 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472375 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472379 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472383 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472386 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472389 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.472416 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:27:47.484538 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472451 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472458 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472460 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472463 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472466 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472468 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472482 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472485 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472487 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472490 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472493 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472496 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472499 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472501 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472504 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472508 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472511 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472514 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472517 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472520 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:47.484976 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472522 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472525 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472527 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472530 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472533 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472536 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472539 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472541 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472544 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472546 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472549 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472552 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472554 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472558 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472560 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472563 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472565 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472568 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472571 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472574 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:47.485808 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472577 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472579 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472582 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472584 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472587 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472590 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472592 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472595 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472597 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472600 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472604 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472607 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472610 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472613 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472615 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472618 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472620 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472626 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472629 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:47.486643 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472633 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472636 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472638 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472641 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472643 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472646 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472650 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472653 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472655 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472658 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472660 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472663 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472665 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472668 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472670 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472673 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472676 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472678 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472680 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:47.487140 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472683 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472686 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472688 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472691 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472693 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472696 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472699 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472701 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472704 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472706 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472708 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472711 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472714 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472717 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472719 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472722 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472724 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472727 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472730 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472733 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:47.487811 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472737 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472740 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472742 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472745 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472747 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472750 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472752 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.472755 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.482324 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.482347 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.482371 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.482378 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482468 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482499 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482506 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482510 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482515 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482520 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482524 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:47.488287 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482528 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482533 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482537 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482541 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482544 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482549 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482553 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482556 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482561 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482565 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482569 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482573 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482577 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482581 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482586 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482590 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482594 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482599 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482603 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:47.488817 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482607 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482612 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482619 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482623 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482628 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482632 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482636 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482640 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482645 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482650 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482654 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482659 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482663 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482668 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482672 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482676 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482680 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482685 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482690 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482694 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:47.489262 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482699 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482703 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482708 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482712 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482717 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482723 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482733 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482738 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482744 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482749 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482753 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482757 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482763 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482768 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482773 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482777 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482781 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482785 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482790 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:47.489831 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482794 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482798 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482803 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482807 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482812 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482817 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482821 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482825 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482830 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482834 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482839 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482843 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482848 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482863 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482868 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482872 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482877 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482882 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482886 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482890 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:47.490296 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482895 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482900 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482904 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482909 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482913 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482917 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482922 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482927 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482932 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482936 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482941 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482945 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482949 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482954 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482958 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482962 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482966 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482973 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482980 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:47.490966 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.482985 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.483031 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483279 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483289 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483295 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483299 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483304 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483317 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483322 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483326 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483330 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483334 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483339 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483343 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483348 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483352 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:47.491431 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483357 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483361 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483365 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483369 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483375 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483379 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483384 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483389 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483393 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483398 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483402 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483407 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483411 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483415 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483419 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483424 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483430 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483437 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:47.491879 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483442 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483447 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483453 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483457 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483461 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483466 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483495 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483501 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483505 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483510 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483514 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483518 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483522 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483526 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483531 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483536 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483541 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483545 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483549 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:47.492609 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483554 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483558 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483562 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483567 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483571 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483576 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483580 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483584 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483589 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483593 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483597 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483602 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483606 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483610 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483615 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483619 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483624 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483628 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483632 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483637 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:47.493376 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483643 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483647 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483651 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483655 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483660 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483664 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483668 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483672 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483676 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483680 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483685 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483689 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483693 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483698 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483702 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483707 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483711 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483716 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483720 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483724 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:47.494208 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483728 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483733 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483738 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483742 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483746 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483751 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483755 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483759 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483763 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483768 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483772 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483779 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483784 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483792 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483796 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.483841 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:27:47.494736 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483895 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483903 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483907 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483912 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483916 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483921 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483925 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483930 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483934 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483938 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483943 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483947 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483952 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483956 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483960 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483967 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483974 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483980 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:47.495124 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483985 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483989 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483994 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.483998 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484002 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484006 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484010 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484015 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484020 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484025 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484029 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484034 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484040 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484044 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484049 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484053 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484057 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484061 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484066 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:47.495663 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484070 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484074 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484078 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484083 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484087 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484091 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484095 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484099 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484104 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484108 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484113 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484117 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484122 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484126 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484130 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484134 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484139 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484143 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484147 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484151 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:47.496134 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484156 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484161 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484165 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484170 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484175 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484179 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484184 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484188 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484193 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484197 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484201 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484205 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484210 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484215 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484219 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484223 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484227 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484232 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484236 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484240 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:47.496617 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484245 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484249 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484254 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484258 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484263 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484267 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484271 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484275 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484280 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484285 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484292 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484298 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484303 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484307 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484312 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484316 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484320 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484325 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484329 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484336 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:47.497175 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484340 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484345 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484349 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484353 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484357 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484362 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484367 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484371 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:47.484375 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.484468 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.487391 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.488570 2573 server.go:1014] "Starting client certificate rotation" Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.488682 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.489657 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.492211 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:27:47.497667 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.494161 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:27:47.507297 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.507280 2573 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:27:47.510992 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.510976 2573 log.go:25] "Validated CRI v1 image API" Nov 02 19:27:47.511931 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.511894 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:27:47.517144 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.517122 2573 fs.go:135] Filesystem UUIDs: map[3ab770dc-49a2-4bf6-939d-61d9ac9e989e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7b653955-db88-4b7d-8c76-f16bd4af0e76:/dev/nvme0n1p4] Nov 02 19:27:47.517216 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.517142 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 19:27:47.524031 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.522647 2573 manager.go:217] Machine: {Timestamp:2026-11-02 19:27:47.521260978 +0000 UTC m=+0.359766851 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099061 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c42087a1b2f0cf32e4527c1c13b2 SystemUUID:ec26c420-87a1-b2f0-cf32-e4527c1c13b2 BootID:134be83f-6f7c-42ff-9df4-f9a419ff198f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e0:b0:3d:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e0:b0:3d:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:0b:5c:c6:c4:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:27:47.524031 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.523561 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:27:47.524031 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.523706 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:27:47.527300 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.527248 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:27:47.527487 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.527299 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-244.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:27:47.528184 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.528173 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:27:47.528215 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.528187 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:27:47.528215 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.528201 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:27:47.528215 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.528213 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:27:47.529951 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.529939 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:27:47.533432 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.533420 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:27:47.533610 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.533600 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:27:47.534514 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.534504 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:27:47.534549 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.534520 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:27:47.534549 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.534532 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:27:47.534549 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.534542 2573 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:27:47.534652 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.534551 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:27:47.534941 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.534929 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:27:47.537486 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.537455 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:27:47.543107 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.543088 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:27:47.544192 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.544178 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:27:47.544253 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.544208 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:27:47.544368 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.544357 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:27:47.544408 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.544373 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:27:47.544408 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.544379 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:27:47.544408 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.544385 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:27:47.544408 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.544391 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:27:47.544408 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.544397 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:27:47.545385 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.545375 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:27:47.545385 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.545384 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:27:47.545461 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.545392 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:27:47.545461 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.545399 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:27:47.545461 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.545407 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:27:47.545461 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.545416 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:27:47.546327 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.546316 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:27:47.546327 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.546325 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:27:47.550160 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.550143 2573 server.go:1263] "Started kubelet" Nov 02 19:27:47.550247 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.550204 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:27:47.550329 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.550277 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:27:47.550382 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.550351 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:27:47.551025 ip-10-0-131-244 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:27:47.551560 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.551541 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:27:47.553588 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.553573 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:27:47.556585 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.556568 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 19:27:47.559189 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.559163 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:27:47.559848 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.559836 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:27:47.559989 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.559949 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-244.us-west-2.compute.internal\" not found" Nov 02 19:27:47.560586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.560552 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:27:47.561070 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.561052 2573 factory.go:55] Registering systemd factory Nov 02 19:27:47.561142 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.561076 2573 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:27:47.561304 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.561292 2573 factory.go:153] Registering CRI-O factory Nov 02 19:27:47.561304 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.561305 2573 factory.go:223] Registration of the crio container factory successfully Nov 02 19:27:47.561416 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.561356 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:27:47.561416 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.561375 2573 factory.go:103] Registering Raw factory Nov 02 19:27:47.561416 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.561388 2573 manager.go:1199] Started watching for new ooms in manager Nov 02 19:27:47.562459 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.562443 2573 manager.go:319] Starting recovery of all containers Nov 02 19:27:47.562562 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.562514 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:27:47.562562 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.562535 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:27:47.562682 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.562669 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:27:47.562682 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.562681 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:27:47.575009 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.574991 2573 manager.go:324] Recovery completed Nov 02 19:27:47.579322 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.579309 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:47.581189 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.581167 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:47.581271 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.581201 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:47.581271 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.581212 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:47.581779 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.581763 2573 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:27:47.581779 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.581774 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:27:47.581779 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.581780 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:47.581962 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.581792 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:27:47.582453 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.582440 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:47.582579 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.582467 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:47.582579 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.582499 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:47.583605 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.583591 2573 policy_none.go:50] "Start" Nov 02 19:27:47.583664 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.583609 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:27:47.583664 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.583628 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:27:47.584984 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.584972 2573 policy_none.go:44] "Start" Nov 02 19:27:47.626325 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.626125 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:27:47.630327 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.630309 2573 manager.go:341] "Starting Device Plugin manager" Nov 02 19:27:47.631091 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.631074 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:27:47.631199 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.631092 2573 server.go:87] "Starting device plugin registration server" Nov 02 19:27:47.631367 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.631356 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:27:47.631417 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.631368 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:27:47.631509 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.631464 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:27:47.631605 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.631592 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:27:47.631605 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.631603 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:27:47.632163 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.632145 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:27:47.632255 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.632192 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-244.us-west-2.compute.internal\" not found" Nov 02 19:27:47.684070 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.684034 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-244.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 19:27:47.684070 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.684036 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-244.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 19:27:47.685233 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.684035 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-244.us-west-2.compute.internal.18e4510cec859d84 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-244.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-244.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:27:47.549937028 +0000 UTC m=+0.388442894,LastTimestamp:2026-11-02 19:27:47.549937028 +0000 UTC m=+0.388442894,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-244.us-west-2.compute.internal,}" Nov 02 19:27:47.691247 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.691214 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:27:47.692428 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.692410 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:27:47.693065 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.693044 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:27:47.693144 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.693088 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:27:47.693144 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.693130 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 19:27:47.731559 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.731482 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:47.733690 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.733671 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:47.733782 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.733707 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:47.733782 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.733718 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:47.733782 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.733753 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.749372 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.749346 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:27:47.749465 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.749414 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6hqsm" Nov 02 19:27:47.793236 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.793188 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal"] Nov 02 19:27:47.793400 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.793291 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:47.794295 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.794279 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:47.794380 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.794310 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:47.794380 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.794320 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:47.795828 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.795814 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:47.795994 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.795971 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.796066 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.796018 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:47.796618 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.796601 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:47.796618 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.796630 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:47.796749 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.796647 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:47.796749 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.796675 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:47.796749 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.796704 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:47.796749 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.796726 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:47.798192 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.798178 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.798240 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.798204 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:47.799043 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.799018 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:47.799139 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.799076 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:47.799139 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.799092 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:47.817617 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.817579 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-244.us-west-2.compute.internal\" not found" node="ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.821568 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:47.821543 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-244.us-west-2.compute.internal\" not found" node="ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.844184 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.844159 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:47.863817 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.863776 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.863817 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.863822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.863972 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.863840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/620b74bb5da064bc73a13b120097bf7c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"620b74bb5da064bc73a13b120097bf7c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.937161 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.937125 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:47.937673 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.937655 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:47.938398 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.938379 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:47.939201 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.939178 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.960965 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.960925 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.964140 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.964115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.964248 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.964226 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.964431 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.964148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.964648 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.964628 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/620b74bb5da064bc73a13b120097bf7c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"620b74bb5da064bc73a13b120097bf7c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.964722 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.964663 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:47.964722 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:47.964684 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/620b74bb5da064bc73a13b120097bf7c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"620b74bb5da064bc73a13b120097bf7c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:48.017897 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.017863 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:27:48.019000 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.018986 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:48.079726 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.079701 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:27:48.121038 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.120996 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:48.125954 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.125558 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:27:48.489481 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.489441 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 19:27:48.490096 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.489699 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:27:48.490096 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.489728 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:27:48.490096 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.489742 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:27:48.491005 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:48.489760 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a15d747bf66324e00b1608078aae7d90-1121cf0d51ffc33d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-244.us-west-2.compute.internal.18e4510cee62e473\": read tcp 10.0.131.244:46112->3.17.51.150:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-244.us-west-2.compute.internal.18e4510cee62e473 default 5445 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-244.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-244.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-244.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:27:47 +0000 UTC,LastTimestamp:2026-11-02 19:27:47.733721762 +0000 UTC m=+0.572227627,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-244.us-west-2.compute.internal,}" Nov 02 19:27:48.536579 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.536524 2573 apiserver.go:52] "Watching apiserver" Nov 02 19:27:48.559652 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.559612 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 19:27:48.754746 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.754589 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:22:48 +0000 UTC" deadline="2028-07-02 03:54:25.942032558 +0000 UTC" Nov 02 19:27:48.754746 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.754651 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14576h26m37.187387253s" Nov 02 19:27:48.804697 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.804663 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:27:48.804999 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.804980 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-cgzbx","openshift-multus/network-metrics-daemon-tq4ks","openshift-network-diagnostics/network-check-target-ssjs9","openshift-ovn-kubernetes/ovnkube-node-8sdfd","kube-system/konnectivity-agent-4wwnx","openshift-dns/node-resolver-zlvtc","openshift-image-registry/node-ca-7bzn9","openshift-multus/multus-dkvbt","openshift-network-operator/iptables-alerter-jhx4c","kube-system/global-pull-secret-syncer-r8qkh","kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8","openshift-cluster-node-tuning-operator/tuned-flwqt"] Nov 02 19:27:48.806799 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.806776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.808172 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.808152 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:48.808296 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:48.808237 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:27:48.809549 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.809531 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:48.809613 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:48.809597 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:27:48.810982 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.810965 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.812416 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.812399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:27:48.814048 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.814029 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:48.815549 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.815521 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:48.817076 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.817062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.818618 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.818605 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:48.820520 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.820506 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:48.820596 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:48.820563 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:27:48.822220 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.822204 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.823638 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.823622 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.862057 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.862034 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:27:48.862200 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.862087 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:27:48.862200 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.862112 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:27:48.862200 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.862178 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:27:48.862294 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.862205 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:27:48.862294 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.862205 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:27:48.862405 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.862391 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bg56f\"" Nov 02 19:27:48.864422 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.864401 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:27:48.864696 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.864680 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:27:48.864784 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.864716 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:27:48.864784 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.864727 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:27:48.864784 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.864722 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:27:48.864784 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.864753 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6bb8t\"" Nov 02 19:27:48.864929 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.864848 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:27:48.865522 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.865506 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4nhx\"" Nov 02 19:27:48.865581 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.865540 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:27:48.865649 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.865638 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:27:48.866982 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.866967 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:27:48.867041 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.866976 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:27:48.869660 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869644 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-kubelet\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.869715 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869669 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-kubelet-config\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:48.869715 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.869715 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z4dh\" (UniqueName: \"kubernetes.io/projected/24100d48-fca9-4735-81a6-a2455bf00804-kube-api-access-5z4dh\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.869814 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-lib-modules\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.869858 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6f7f\" (UniqueName: \"kubernetes.io/projected/53fc9047-2f91-4806-acfd-d23b1b0cbc71-kube-api-access-s6f7f\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:48.869858 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-bin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.869916 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-hostroot\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.869968 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsg8n\" (UniqueName: \"kubernetes.io/projected/2602c042-a534-4767-8bb3-e5d2b93b3a62-kube-api-access-vsg8n\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:48.869968 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.869945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:48.870060 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2602c042-a534-4767-8bb3-e5d2b93b3a62-iptables-alerter-script\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:48.870102 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-slash\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.870139 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.870182 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870135 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5bhk\" (UniqueName: \"kubernetes.io/projected/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-kube-api-access-d5bhk\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.870220 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-ovn\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.870220 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6f4\" (UniqueName: \"kubernetes.io/projected/f31b231b-c403-428b-a1a3-f516644ac9e5-kube-api-access-sb6f4\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:48.870297 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-etc-kubernetes\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.870297 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:48.870366 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-node-log\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.870366 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-bin\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.870447 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28kv4\" (UniqueName: \"kubernetes.io/projected/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-kube-api-access-28kv4\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.870447 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870396 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-multus\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.870447 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-multus-certs\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.870590 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-agent-certs\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:27:48.870590 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-modprobe-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.870590 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870571 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.870702 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870631 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-system-cni-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.870702 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.870702 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870685 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53fc9047-2f91-4806-acfd-d23b1b0cbc71-serviceca\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:48.870821 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.870821 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870744 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-conf\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.870821 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-systemd\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.870916 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-var-lib-kubelet\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.870916 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-host\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.870916 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-env-overrides\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.871060 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.871060 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870963 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-k8s-cni-cncf-io\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.871060 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.870990 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9lvp\" (UniqueName: \"kubernetes.io/projected/19ec87c5-1d05-4829-9a6d-cddcd0f84607-kube-api-access-n9lvp\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:48.871060 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdz2\" (UniqueName: \"kubernetes.io/projected/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-kube-api-access-hhdz2\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.871060 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-systemd-units\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-etc-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31b231b-c403-428b-a1a3-f516644ac9e5-tmp-dir\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-cni-binary-copy\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-socket-dir-parent\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-conf-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-dbus\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-cnibin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-run\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-log-socket\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.871330 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-netns\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2602c042-a534-4767-8bb3-e5d2b93b3a62-host-slash\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-socket-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871468 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-netd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53fc9047-2f91-4806-acfd-d23b1b0cbc71-host\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871525 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-kubelet\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-registration-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5p9\" (UniqueName: \"kubernetes.io/projected/42a08de9-3178-4d75-87b7-0cea7a813dd3-kube-api-access-tq5p9\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-konnectivity-ca\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-kubernetes\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871654 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-device-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871691 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-sys-fs\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysconfig\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-multus-daemon-config\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.872029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871739 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-netns\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-var-lib-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-config\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-script-lib\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871828 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-systemd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovn-node-metrics-cert\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f31b231b-c403-428b-a1a3-f516644ac9e5-hosts-file\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-tmp\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871915 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cnibin\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871948 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-system-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871964 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-sys\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.872614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.871979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-tuned\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.913000 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.912971 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:27:48.913141 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913030 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wxdrr\"" Nov 02 19:27:48.913141 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913084 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:27:48.913141 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913126 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:27:48.913279 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913153 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:27:48.913279 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913180 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gsgfx\"" Nov 02 19:27:48.913279 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913194 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nnqx8\"" Nov 02 19:27:48.913279 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913270 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lnn7q\"" Nov 02 19:27:48.913450 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913380 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:27:48.913519 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913505 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:27:48.913519 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913513 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:27:48.913702 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913687 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:27:48.913758 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913746 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:27:48.913829 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913817 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:27:48.913862 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913832 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9t85s\"" Nov 02 19:27:48.913862 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913842 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:27:48.913923 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913905 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:27:48.913923 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.913911 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sp8wk\"" Nov 02 19:27:48.915064 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.915045 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:27:48.915102 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.915076 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-snq44" Nov 02 19:27:48.972302 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6f7f\" (UniqueName: \"kubernetes.io/projected/53fc9047-2f91-4806-acfd-d23b1b0cbc71-kube-api-access-s6f7f\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:48.972302 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972303 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-bin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.972542 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-hostroot\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.972542 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsg8n\" (UniqueName: \"kubernetes.io/projected/2602c042-a534-4767-8bb3-e5d2b93b3a62-kube-api-access-vsg8n\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:48.972542 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:48.972542 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2602c042-a534-4767-8bb3-e5d2b93b3a62-iptables-alerter-script\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:48.972542 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972397 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-hostroot\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.972542 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-slash\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.972542 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972397 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-bin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.972542 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.972542 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:48.972535 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972544 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-slash\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:48.972624 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret podName:13da2219-8d3c-4b28-b6b0-38b51415de22 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:49.472599211 +0000 UTC m=+2.311105088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret") pod "global-pull-secret-syncer-r8qkh" (UID: "13da2219-8d3c-4b28-b6b0-38b51415de22") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5bhk\" (UniqueName: \"kubernetes.io/projected/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-kube-api-access-d5bhk\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-ovn\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6f4\" (UniqueName: \"kubernetes.io/projected/f31b231b-c403-428b-a1a3-f516644ac9e5-kube-api-access-sb6f4\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972725 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-etc-kubernetes\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972764 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-ovn\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972794 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-etc-kubernetes\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-node-log\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-bin\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28kv4\" (UniqueName: \"kubernetes.io/projected/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-kube-api-access-28kv4\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972869 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-multus\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972873 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-node-log\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.973010 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972990 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-multus\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973310 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2602c042-a534-4767-8bb3-e5d2b93b3a62-iptables-alerter-script\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-multus-certs\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-agent-certs\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-modprobe-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973491 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-multus-certs\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.972893 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-bin\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-system-cni-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973636 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-system-cni-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973691 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973694 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-modprobe-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.973733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53fc9047-2f91-4806-acfd-d23b1b0cbc71-serviceca\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:48.974353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.974353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973843 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:27:48.974353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973917 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-conf\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.974353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.973956 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.974353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-conf\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.974353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-systemd\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.974353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974104 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-var-lib-kubelet\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974404 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53fc9047-2f91-4806-acfd-d23b1b0cbc71-serviceca\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974476 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-systemd\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974524 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-host\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974625 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-var-lib-kubelet\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-env-overrides\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974673 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-host\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-k8s-cni-cncf-io\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.974798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9lvp\" (UniqueName: \"kubernetes.io/projected/19ec87c5-1d05-4829-9a6d-cddcd0f84607-kube-api-access-n9lvp\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdz2\" (UniqueName: \"kubernetes.io/projected/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-kube-api-access-hhdz2\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974860 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-systemd-units\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-etc-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974900 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974924 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31b231b-c403-428b-a1a3-f516644ac9e5-tmp-dir\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-cni-binary-copy\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.974994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-socket-dir-parent\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-conf-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-dbus\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975124 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-cnibin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-run\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975194 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-log-socket\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975204 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-systemd-units\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975244 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-env-overrides\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975251 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-etc-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.975284 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-netns\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2602c042-a534-4767-8bb3-e5d2b93b3a62-host-slash\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975334 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-conf-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975360 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2602c042-a534-4767-8bb3-e5d2b93b3a62-host-slash\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-socket-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-netd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975496 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53fc9047-2f91-4806-acfd-d23b1b0cbc71-host\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975547 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-dbus\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975586 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31b231b-c403-428b-a1a3-f516644ac9e5-tmp-dir\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975619 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-cnibin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975634 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-kubelet\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975565 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-kubelet\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975669 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-k8s-cni-cncf-io\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-registration-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975717 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-netns\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975737 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5p9\" (UniqueName: \"kubernetes.io/projected/42a08de9-3178-4d75-87b7-0cea7a813dd3-kube-api-access-tq5p9\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-konnectivity-ca\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:27:48.976177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975773 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-run\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-kubernetes\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-device-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-sys-fs\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysconfig\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975982 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-multus-daemon-config\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-netns\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-var-lib-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-config\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-script-lib\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976204 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-systemd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovn-node-metrics-cert\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976285 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-netd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975462 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-socket-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.976988 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-cni-binary-copy\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976352 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-registration-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976369 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53fc9047-2f91-4806-acfd-d23b1b0cbc71-host\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.975497 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-socket-dir-parent\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f31b231b-c403-428b-a1a3-f516644ac9e5-hosts-file\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976400 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-log-socket\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-tmp\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976418 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f31b231b-c403-428b-a1a3-f516644ac9e5-hosts-file\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cnibin\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976486 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-system-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-sys\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-tuned\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-kubelet\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976677 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-kubelet-config\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976730 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z4dh\" (UniqueName: \"kubernetes.io/projected/24100d48-fca9-4735-81a6-a2455bf00804-kube-api-access-5z4dh\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.977586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976783 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-var-lib-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-lib-modules\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976954 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-lib-modules\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.976996 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-kubernetes\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.977044 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-device-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.977085 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-sys-fs\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.977127 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysconfig\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.977149 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-konnectivity-ca\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.977229 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.977714 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-netns\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.977967 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-sys\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.978025 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-system-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.978075 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-kubelet\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.978079 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-kubelet-config\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.978118 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.978148 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cnibin\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.978323 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.978225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-systemd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.979038 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.978328 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.979038 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:48.978360 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:48.979038 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.978402 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-config\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.979038 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:48.978427 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs podName:19ec87c5-1d05-4829-9a6d-cddcd0f84607 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:49.478407684 +0000 UTC m=+2.316913553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs") pod "network-metrics-daemon-tq4ks" (UID: "19ec87c5-1d05-4829-9a6d-cddcd0f84607") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:48.979038 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.978469 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-multus-daemon-config\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:48.979432 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.979398 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:48.979865 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.979767 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-agent-certs\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:27:48.980307 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.980285 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-script-lib\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.981024 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.981006 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-tmp\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:48.981105 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.981054 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovn-node-metrics-cert\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:48.981937 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:48.981898 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-tuned\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:49.029756 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.029670 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:49.029756 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.029705 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:49.029756 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.029718 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fjqwf for pod openshift-network-diagnostics/network-check-target-ssjs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:49.029999 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.029795 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf podName:c967790c-1a44-4339-aa6e-8becad9f4474 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:49.529771277 +0000 UTC m=+2.368277152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqwf" (UniqueName: "kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf") pod "network-check-target-ssjs9" (UID: "c967790c-1a44-4339-aa6e-8becad9f4474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:49.032509 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.032479 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6f4\" (UniqueName: \"kubernetes.io/projected/f31b231b-c403-428b-a1a3-f516644ac9e5-kube-api-access-sb6f4\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:49.032859 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.032792 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28kv4\" (UniqueName: \"kubernetes.io/projected/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-kube-api-access-28kv4\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:49.033003 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.032980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6f7f\" (UniqueName: \"kubernetes.io/projected/53fc9047-2f91-4806-acfd-d23b1b0cbc71-kube-api-access-s6f7f\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:49.033003 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.032993 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsg8n\" (UniqueName: \"kubernetes.io/projected/2602c042-a534-4767-8bb3-e5d2b93b3a62-kube-api-access-vsg8n\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:49.033202 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.033181 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdz2\" (UniqueName: \"kubernetes.io/projected/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-kube-api-access-hhdz2\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:49.033489 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.033464 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5bhk\" (UniqueName: \"kubernetes.io/projected/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-kube-api-access-d5bhk\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:49.034684 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.034662 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9lvp\" (UniqueName: \"kubernetes.io/projected/19ec87c5-1d05-4829-9a6d-cddcd0f84607-kube-api-access-n9lvp\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:49.034801 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.034779 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5p9\" (UniqueName: \"kubernetes.io/projected/42a08de9-3178-4d75-87b7-0cea7a813dd3-kube-api-access-tq5p9\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:49.034867 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.034851 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z4dh\" (UniqueName: \"kubernetes.io/projected/24100d48-fca9-4735-81a6-a2455bf00804-kube-api-access-5z4dh\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:27:49.151214 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.151181 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:27:49.170661 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.170631 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:27:49.174529 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.174496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:27:49.195775 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.195725 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:27:49.206143 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.206111 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:27:49.223935 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.223906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dkvbt" Nov 02 19:27:49.241049 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.241017 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:27:49.250041 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.250014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:27:49.251114 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.251091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:27:49.286948 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:49.286859 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620b74bb5da064bc73a13b120097bf7c.slice/crio-fb59f3c761721d25c75ceac6d48db30c50ac93b0fe5361d3b199e33e313c4799 WatchSource:0}: Error finding container fb59f3c761721d25c75ceac6d48db30c50ac93b0fe5361d3b199e33e313c4799: Status 404 returned error can't find the container with id fb59f3c761721d25c75ceac6d48db30c50ac93b0fe5361d3b199e33e313c4799 Nov 02 19:27:49.287601 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:49.287572 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd5b763ddc1b8b385e585752b67b52f.slice/crio-010fac1e5b127921d19fae7d389c4501ef76dcada819b5030e50cfed167af910 WatchSource:0}: Error finding container 010fac1e5b127921d19fae7d389c4501ef76dcada819b5030e50cfed167af910: Status 404 returned error can't find the container with id 010fac1e5b127921d19fae7d389c4501ef76dcada819b5030e50cfed167af910 Nov 02 19:27:49.288285 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:49.288257 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a08de9_3178_4d75_87b7_0cea7a813dd3.slice/crio-2573d463444e98c321960ffca385dbadae267636744405fc811321d91d50db65 WatchSource:0}: Error finding container 2573d463444e98c321960ffca385dbadae267636744405fc811321d91d50db65: Status 404 returned error can't find the container with id 2573d463444e98c321960ffca385dbadae267636744405fc811321d91d50db65 Nov 02 19:27:49.288928 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:49.288906 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6d9c0b_33ed_4ea1_9533_4ce89780bb97.slice/crio-2016119f67c80f3f42243bdc3fc2c6a8cdae4e1c9f2ce05d5e037490e92ca015 WatchSource:0}: Error finding container 2016119f67c80f3f42243bdc3fc2c6a8cdae4e1c9f2ce05d5e037490e92ca015: Status 404 returned error can't find the container with id 2016119f67c80f3f42243bdc3fc2c6a8cdae4e1c9f2ce05d5e037490e92ca015 Nov 02 19:27:49.293597 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.293579 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:27:49.435351 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.435322 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:49.480637 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.480592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:49.480838 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.480652 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:49.480838 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.480730 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:49.480838 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.480795 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:49.480838 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.480802 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs podName:19ec87c5-1d05-4829-9a6d-cddcd0f84607 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:50.4807864 +0000 UTC m=+3.319292254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs") pod "network-metrics-daemon-tq4ks" (UID: "19ec87c5-1d05-4829-9a6d-cddcd0f84607") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:49.481007 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.480862 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret podName:13da2219-8d3c-4b28-b6b0-38b51415de22 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:50.480845939 +0000 UTC m=+3.319351801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret") pod "global-pull-secret-syncer-r8qkh" (UID: "13da2219-8d3c-4b28-b6b0-38b51415de22") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:49.494609 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.494574 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:49.582033 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.581947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:49.582201 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.582071 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:49.582201 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.582088 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:49.582201 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.582099 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fjqwf for pod openshift-network-diagnostics/network-check-target-ssjs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:49.582201 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:49.582150 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf podName:c967790c-1a44-4339-aa6e-8becad9f4474 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:50.582136655 +0000 UTC m=+3.420642524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqwf" (UniqueName: "kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf") pod "network-check-target-ssjs9" (UID: "c967790c-1a44-4339-aa6e-8becad9f4474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:49.698139 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.698074 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"1cd5b763ddc1b8b385e585752b67b52f","Type":"ContainerStarted","Data":"010fac1e5b127921d19fae7d389c4501ef76dcada819b5030e50cfed167af910"} Nov 02 19:27:49.699273 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.699251 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"620b74bb5da064bc73a13b120097bf7c","Type":"ContainerStarted","Data":"fb59f3c761721d25c75ceac6d48db30c50ac93b0fe5361d3b199e33e313c4799"} Nov 02 19:27:49.700337 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.700315 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" event={"ID":"42a08de9-3178-4d75-87b7-0cea7a813dd3","Type":"ContainerStarted","Data":"2573d463444e98c321960ffca385dbadae267636744405fc811321d91d50db65"} Nov 02 19:27:49.701417 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.701397 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flwqt" event={"ID":"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97","Type":"ContainerStarted","Data":"2016119f67c80f3f42243bdc3fc2c6a8cdae4e1c9f2ce05d5e037490e92ca015"} Nov 02 19:27:49.908966 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.908880 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:49.917687 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.917648 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:22:49 +0000 UTC" deadline="2028-03-30 23:48:33.591267524 +0000 UTC" Nov 02 19:27:49.917687 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:49.917682 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12340h20m43.673588393s" Nov 02 19:27:50.074359 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:50.074318 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9559a8_3f33_4ea2_b478_8b9474d99ed7.slice/crio-47a581c4ec21a6c6ed42132dfa90f4296f0e49de45b8537352e115e71a010a86 WatchSource:0}: Error finding container 47a581c4ec21a6c6ed42132dfa90f4296f0e49de45b8537352e115e71a010a86: Status 404 returned error can't find the container with id 47a581c4ec21a6c6ed42132dfa90f4296f0e49de45b8537352e115e71a010a86 Nov 02 19:27:50.088465 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:50.088438 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fc00c2_cfbb_4d44_867e_0b9b7c61cd0f.slice/crio-125e3cca30e0c0415432b25dca84e50a35329edce80ff1b0b3d0ddc3690bf18b WatchSource:0}: Error finding container 125e3cca30e0c0415432b25dca84e50a35329edce80ff1b0b3d0ddc3690bf18b: Status 404 returned error can't find the container with id 125e3cca30e0c0415432b25dca84e50a35329edce80ff1b0b3d0ddc3690bf18b Nov 02 19:27:50.102158 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:50.102131 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24100d48_fca9_4735_81a6_a2455bf00804.slice/crio-dd2a4d606f0a0c4df49062b4adcf1ff50aa7c43406c4a7c6b0ff5e023a05051a WatchSource:0}: Error finding container dd2a4d606f0a0c4df49062b4adcf1ff50aa7c43406c4a7c6b0ff5e023a05051a: Status 404 returned error can't find the container with id dd2a4d606f0a0c4df49062b4adcf1ff50aa7c43406c4a7c6b0ff5e023a05051a Nov 02 19:27:50.108604 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:50.108574 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fc9047_2f91_4806_acfd_d23b1b0cbc71.slice/crio-9d733afefb0412f2afa49be26fbc7502b827d31df450cccad406a2c401b73a9a WatchSource:0}: Error finding container 9d733afefb0412f2afa49be26fbc7502b827d31df450cccad406a2c401b73a9a: Status 404 returned error can't find the container with id 9d733afefb0412f2afa49be26fbc7502b827d31df450cccad406a2c401b73a9a Nov 02 19:27:50.111806 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:50.111783 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31b231b_c403_428b_a1a3_f516644ac9e5.slice/crio-bd10fc894d0a3082e0098d4444e6cd275aa026871f299331bcc904c7ccd6ea53 WatchSource:0}: Error finding container bd10fc894d0a3082e0098d4444e6cd275aa026871f299331bcc904c7ccd6ea53: Status 404 returned error can't find the container with id bd10fc894d0a3082e0098d4444e6cd275aa026871f299331bcc904c7ccd6ea53 Nov 02 19:27:50.116158 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:50.116134 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71e26dd_fe22_44a6_9b04_bf30e4aef71a.slice/crio-2d9a6ad973d3cc9f414d9443c94cbd1984d66af12b8401a86ade6ca6dd2388b8 WatchSource:0}: Error finding container 2d9a6ad973d3cc9f414d9443c94cbd1984d66af12b8401a86ade6ca6dd2388b8: Status 404 returned error can't find the container with id 2d9a6ad973d3cc9f414d9443c94cbd1984d66af12b8401a86ade6ca6dd2388b8 Nov 02 19:27:50.139470 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:27:50.139436 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2602c042_a534_4767_8bb3_e5d2b93b3a62.slice/crio-88f4fa50454796aaf822bc3612d51da1c57f8ba5155b13a1e488fd4da2a95df1 WatchSource:0}: Error finding container 88f4fa50454796aaf822bc3612d51da1c57f8ba5155b13a1e488fd4da2a95df1: Status 404 returned error can't find the container with id 88f4fa50454796aaf822bc3612d51da1c57f8ba5155b13a1e488fd4da2a95df1 Nov 02 19:27:50.490696 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.490623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:50.490696 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.490686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:50.490981 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.490804 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:50.490981 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.490849 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:50.490981 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.490867 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret podName:13da2219-8d3c-4b28-b6b0-38b51415de22 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:52.490848399 +0000 UTC m=+5.329354273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret") pod "global-pull-secret-syncer-r8qkh" (UID: "13da2219-8d3c-4b28-b6b0-38b51415de22") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:50.490981 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.490946 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs podName:19ec87c5-1d05-4829-9a6d-cddcd0f84607 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:52.4909225 +0000 UTC m=+5.329428374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs") pod "network-metrics-daemon-tq4ks" (UID: "19ec87c5-1d05-4829-9a6d-cddcd0f84607") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:50.591675 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.591612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:50.592339 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.591820 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:50.592339 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.591840 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:50.592339 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.591853 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fjqwf for pod openshift-network-diagnostics/network-check-target-ssjs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:50.592339 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.591918 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf podName:c967790c-1a44-4339-aa6e-8becad9f4474 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:52.591899023 +0000 UTC m=+5.430404885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqwf" (UniqueName: "kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf") pod "network-check-target-ssjs9" (UID: "c967790c-1a44-4339-aa6e-8becad9f4474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:50.693362 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.693321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:50.693589 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.693462 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:27:50.693589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.693582 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:50.693719 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.693671 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:27:50.693778 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.693740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:50.693836 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:50.693825 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:27:50.706006 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.705967 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhx4c" event={"ID":"2602c042-a534-4767-8bb3-e5d2b93b3a62","Type":"ContainerStarted","Data":"88f4fa50454796aaf822bc3612d51da1c57f8ba5155b13a1e488fd4da2a95df1"} Nov 02 19:27:50.707565 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.707518 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wwnx" event={"ID":"d71e26dd-fe22-44a6-9b04-bf30e4aef71a","Type":"ContainerStarted","Data":"2d9a6ad973d3cc9f414d9443c94cbd1984d66af12b8401a86ade6ca6dd2388b8"} Nov 02 19:27:50.709325 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.709294 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zlvtc" event={"ID":"f31b231b-c403-428b-a1a3-f516644ac9e5","Type":"ContainerStarted","Data":"bd10fc894d0a3082e0098d4444e6cd275aa026871f299331bcc904c7ccd6ea53"} Nov 02 19:27:50.710881 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.710782 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dkvbt" event={"ID":"24100d48-fca9-4735-81a6-a2455bf00804","Type":"ContainerStarted","Data":"dd2a4d606f0a0c4df49062b4adcf1ff50aa7c43406c4a7c6b0ff5e023a05051a"} Nov 02 19:27:50.714370 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.714341 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7bzn9" event={"ID":"53fc9047-2f91-4806-acfd-d23b1b0cbc71","Type":"ContainerStarted","Data":"9d733afefb0412f2afa49be26fbc7502b827d31df450cccad406a2c401b73a9a"} Nov 02 19:27:50.716124 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.716098 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" event={"ID":"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f","Type":"ContainerStarted","Data":"125e3cca30e0c0415432b25dca84e50a35329edce80ff1b0b3d0ddc3690bf18b"} Nov 02 19:27:50.717818 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.717794 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" event={"ID":"9d9559a8-3f33-4ea2-b478-8b9474d99ed7","Type":"ContainerStarted","Data":"47a581c4ec21a6c6ed42132dfa90f4296f0e49de45b8537352e115e71a010a86"} Nov 02 19:27:50.918381 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.918225 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:22:49 +0000 UTC" deadline="2028-04-07 06:04:32.083407194 +0000 UTC" Nov 02 19:27:50.918381 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:50.918284 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12514h36m41.165126119s" Nov 02 19:27:52.511028 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:52.510934 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:52.511028 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:52.511010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:52.511638 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.511182 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:52.511638 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.511250 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret podName:13da2219-8d3c-4b28-b6b0-38b51415de22 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:56.511231395 +0000 UTC m=+9.349737255 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret") pod "global-pull-secret-syncer-r8qkh" (UID: "13da2219-8d3c-4b28-b6b0-38b51415de22") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:52.511638 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.511338 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:52.511638 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.511378 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs podName:19ec87c5-1d05-4829-9a6d-cddcd0f84607 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:56.511365296 +0000 UTC m=+9.349871157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs") pod "network-metrics-daemon-tq4ks" (UID: "19ec87c5-1d05-4829-9a6d-cddcd0f84607") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:52.613828 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:52.611800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:52.613828 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.611995 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:52.613828 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.612028 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:52.613828 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.612043 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fjqwf for pod openshift-network-diagnostics/network-check-target-ssjs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:52.613828 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.612109 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf podName:c967790c-1a44-4339-aa6e-8becad9f4474 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:56.612087268 +0000 UTC m=+9.450593128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqwf" (UniqueName: "kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf") pod "network-check-target-ssjs9" (UID: "c967790c-1a44-4339-aa6e-8becad9f4474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:52.693666 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:52.693627 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:52.693889 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.693789 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:27:52.693889 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:52.693858 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:52.694016 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.693931 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:27:52.694016 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:52.693979 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:52.694171 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:52.694054 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:27:54.694478 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:54.693865 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:54.694478 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:54.694077 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:27:54.694478 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:54.694166 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:54.694478 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:54.694261 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:27:54.694478 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:54.694320 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:54.694478 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:54.694390 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:27:54.759861 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:54.758816 2573 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="603a22711f09a86054bd7b3a3d56f59f794c5cf9980b11df535d6679ef6bb254" exitCode=0 Nov 02 19:27:54.759861 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:54.758918 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" event={"ID":"9d9559a8-3f33-4ea2-b478-8b9474d99ed7","Type":"ContainerDied","Data":"603a22711f09a86054bd7b3a3d56f59f794c5cf9980b11df535d6679ef6bb254"} Nov 02 19:27:54.766085 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:54.765886 2573 generic.go:356] "Generic (PLEG): container finished" podID="1cd5b763ddc1b8b385e585752b67b52f" containerID="ba1a05b137db964becc0a5e982de2804e00b44d4a26d19ea265058b9b28ee3f7" exitCode=0 Nov 02 19:27:54.766085 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:54.765994 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"1cd5b763ddc1b8b385e585752b67b52f","Type":"ContainerDied","Data":"ba1a05b137db964becc0a5e982de2804e00b44d4a26d19ea265058b9b28ee3f7"} Nov 02 19:27:54.783444 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:54.783407 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"620b74bb5da064bc73a13b120097bf7c","Type":"ContainerStarted","Data":"fba3809ff8c3b599dc404cea8b1d4291913afc20d7e9295d80d9fe2aeb2e7139"} Nov 02 19:27:55.090852 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:55.090783 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" podStartSLOduration=7.090762626 podStartE2EDuration="7.090762626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:55.089814665 +0000 UTC m=+7.928320542" watchObservedRunningTime="2026-11-02 19:27:55.090762626 +0000 UTC m=+7.929268505" Nov 02 19:27:55.790194 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:55.790147 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"1cd5b763ddc1b8b385e585752b67b52f","Type":"ContainerStarted","Data":"c4ec7183a50381e75180d2f0abedb05b25943466ba8754c9e1bf05673f0c1983"} Nov 02 19:27:55.794046 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:55.794012 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wwnx" event={"ID":"d71e26dd-fe22-44a6-9b04-bf30e4aef71a","Type":"ContainerStarted","Data":"d6bfba89b9c4f75f9c583b51fbcc248b9e87d2659c93f5a89841fb20539f90f9"} Nov 02 19:27:55.799433 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:55.799397 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" event={"ID":"42a08de9-3178-4d75-87b7-0cea7a813dd3","Type":"ContainerStarted","Data":"b5bb08fc464e2985625c2010dcaec5547a26622a522227fa24a0b43c20fe9f47"} Nov 02 19:27:56.026426 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:56.026358 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" podStartSLOduration=8.026336428 podStartE2EDuration="8.026336428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:55.910199472 +0000 UTC m=+8.748705351" watchObservedRunningTime="2026-11-02 19:27:56.026336428 +0000 UTC m=+8.864842307" Nov 02 19:27:56.549344 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:56.549304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:56.549574 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:56.549374 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:56.549639 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.549595 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:56.549700 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.549672 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret podName:13da2219-8d3c-4b28-b6b0-38b51415de22 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.54965225 +0000 UTC m=+17.388158106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret") pod "global-pull-secret-syncer-r8qkh" (UID: "13da2219-8d3c-4b28-b6b0-38b51415de22") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:56.549785 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.549762 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:56.549844 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.549809 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs podName:19ec87c5-1d05-4829-9a6d-cddcd0f84607 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.549796158 +0000 UTC m=+17.388302013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs") pod "network-metrics-daemon-tq4ks" (UID: "19ec87c5-1d05-4829-9a6d-cddcd0f84607") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:56.649869 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:56.649818 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:56.650066 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.650035 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:56.650066 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.650058 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:56.650178 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.650072 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fjqwf for pod openshift-network-diagnostics/network-check-target-ssjs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:56.650178 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.650132 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf podName:c967790c-1a44-4339-aa6e-8becad9f4474 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.65011404 +0000 UTC m=+17.488619912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqwf" (UniqueName: "kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf") pod "network-check-target-ssjs9" (UID: "c967790c-1a44-4339-aa6e-8becad9f4474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:56.693647 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:56.693609 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:56.693820 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.693760 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:27:56.693905 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:56.693851 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:56.693958 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.693939 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:27:56.694011 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:56.694000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:56.694111 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:56.694083 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:27:56.805812 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:56.805722 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7bzn9" event={"ID":"53fc9047-2f91-4806-acfd-d23b1b0cbc71","Type":"ContainerStarted","Data":"f1d771a2ead772d55891cab0d07aca99354f9304d066a781ef725c43c5fd2b68"} Nov 02 19:27:56.922099 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:56.922026 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4wwnx" podStartSLOduration=3.892454201 podStartE2EDuration="8.922003036s" totalImagesPullingTime="5.029548835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:56.025828911 +0000 UTC m=+8.864334791" watchObservedRunningTime="2026-11-02 19:27:56.922003036 +0000 UTC m=+9.760508916" Nov 02 19:27:57.786944 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:57.786910 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:27:57.819682 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:57.819465 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" event={"ID":"42a08de9-3178-4d75-87b7-0cea7a813dd3","Type":"ContainerStarted","Data":"bc8d6edee5a53f117da160cb9aa2ae3abc3ebfa68eafd638f70bbab0994b9f2b"} Nov 02 19:27:58.637872 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:58.637730 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:27:57.786938789Z","UUID":"97e11d77-a3ba-4a36-aeab-a645c62d2742","Handler":null,"Name":"","Endpoint":""} Nov 02 19:27:58.639964 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:58.639938 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:27:58.640157 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:58.639975 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:27:58.693827 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:58.693790 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:27:58.694095 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:58.693949 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:27:58.694469 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:58.694443 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:27:58.694616 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:58.694593 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:27:58.694685 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:58.694668 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:27:58.694764 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:27:58.694744 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:27:59.826512 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:59.826256 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" event={"ID":"42a08de9-3178-4d75-87b7-0cea7a813dd3","Type":"ContainerStarted","Data":"a9db49987736dff97cc7beb3d5b265b5e754f6efb247f7a3559b4793f81e6891"} Nov 02 19:27:59.830570 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:59.830535 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" event={"ID":"9d9559a8-3f33-4ea2-b478-8b9474d99ed7","Type":"ContainerStarted","Data":"76afb620a7c4e40917c60ebeac7cb6284680d613289493713e2d5cb967e63d18"} Nov 02 19:27:59.835468 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:59.835430 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dkvbt" event={"ID":"24100d48-fca9-4735-81a6-a2455bf00804","Type":"ContainerStarted","Data":"7fb5235a8ac1818b877cdb260469cd9a0aaf1a635f63af725220b08f226e81ee"} Nov 02 19:27:59.965596 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:59.965539 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" podStartSLOduration=2.028728928 podStartE2EDuration="11.965507808s" totalImagesPullingTime="9.93677888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:59.965186954 +0000 UTC m=+12.803692833" watchObservedRunningTime="2026-11-02 19:27:59.965507808 +0000 UTC m=+12.804013686" Nov 02 19:27:59.966220 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:27:59.966188 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7bzn9" podStartSLOduration=6.093935268 podStartE2EDuration="11.966177778s" totalImagesPullingTime="5.87224251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:56.922630561 +0000 UTC m=+9.761136438" watchObservedRunningTime="2026-11-02 19:27:59.966177778 +0000 UTC m=+12.804683702" Nov 02 19:28:00.049103 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.049065 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:28:00.050340 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.050275 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:28:00.083983 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.083930 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dkvbt" podStartSLOduration=3.082864304 podStartE2EDuration="12.083909916s" totalImagesPullingTime="9.001045612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:00.083271411 +0000 UTC m=+12.921777306" watchObservedRunningTime="2026-11-02 19:28:00.083909916 +0000 UTC m=+12.922415797" Nov 02 19:28:00.693647 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.693618 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:00.693647 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.693650 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:00.693861 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:00.693734 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:00.693861 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.693797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:00.693968 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:00.693896 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:00.694011 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:00.693959 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:00.840099 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.839695 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhx4c" event={"ID":"2602c042-a534-4767-8bb3-e5d2b93b3a62","Type":"ContainerStarted","Data":"4cd937e8c0f28f48b437effa81eacb88e329167e751e3cd0aaaaf45e46cddd31"} Nov 02 19:28:00.841132 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.841107 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zlvtc" event={"ID":"f31b231b-c403-428b-a1a3-f516644ac9e5","Type":"ContainerStarted","Data":"296f0492f388a76ab8176c3edbca712596e15e8fe0e2ae3b83fcefe92b26c249"} Nov 02 19:28:00.843335 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.843311 2573 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="76afb620a7c4e40917c60ebeac7cb6284680d613289493713e2d5cb967e63d18" exitCode=0 Nov 02 19:28:00.843448 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.843409 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" event={"ID":"9d9559a8-3f33-4ea2-b478-8b9474d99ed7","Type":"ContainerDied","Data":"76afb620a7c4e40917c60ebeac7cb6284680d613289493713e2d5cb967e63d18"} Nov 02 19:28:00.844129 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.844107 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:28:00.844724 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.844689 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:28:00.947427 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:00.947302 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jhx4c" podStartSLOduration=3.986820445 podStartE2EDuration="12.947283368s" totalImagesPullingTime="8.960462923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:00.946798185 +0000 UTC m=+13.785304083" watchObservedRunningTime="2026-11-02 19:28:00.947283368 +0000 UTC m=+13.785789244" Nov 02 19:28:01.277070 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:01.276960 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zlvtc" podStartSLOduration=3.312441531 podStartE2EDuration="13.276939935s" totalImagesPullingTime="9.964498404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:01.276305519 +0000 UTC m=+14.114811393" watchObservedRunningTime="2026-11-02 19:28:01.276939935 +0000 UTC m=+14.115445811" Nov 02 19:28:01.847076 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:01.846809 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flwqt" event={"ID":"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97","Type":"ContainerStarted","Data":"84803735c10cef3cb02620005fd904957ed17020e96c4c627b24ab3cd072e8eb"} Nov 02 19:28:01.955957 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:01.955892 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-flwqt" podStartSLOduration=2.221583245 podStartE2EDuration="13.955871248s" totalImagesPullingTime="11.734288003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:01.955707219 +0000 UTC m=+14.794213098" watchObservedRunningTime="2026-11-02 19:28:01.955871248 +0000 UTC m=+14.794377118" Nov 02 19:28:02.670544 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:02.670437 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:02.694342 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:02.694304 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:02.694342 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:02.694341 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:02.694614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:02.694353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:02.694614 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:02.694447 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:02.694614 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:02.694538 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:02.694731 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:02.694635 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:02.850926 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:02.850889 2573 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="6210a725dbe4960fcf103832e35aea8b01966d3cbb304a520c4ed19e40aaa242" exitCode=0 Nov 02 19:28:02.851533 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:02.850984 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" event={"ID":"9d9559a8-3f33-4ea2-b478-8b9474d99ed7","Type":"ContainerDied","Data":"6210a725dbe4960fcf103832e35aea8b01966d3cbb304a520c4ed19e40aaa242"} Nov 02 19:28:02.970021 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:02.969900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:03.183063 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:03.183029 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:04.075842 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.075627 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:04.269025 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.268987 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:04.617059 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.617006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:04.617296 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.617071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:04.617296 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.617151 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:28:04.617296 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.617206 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:28:04.617296 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.617229 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs podName:19ec87c5-1d05-4829-9a6d-cddcd0f84607 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:20.617211659 +0000 UTC m=+33.455717523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs") pod "network-metrics-daemon-tq4ks" (UID: "19ec87c5-1d05-4829-9a6d-cddcd0f84607") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:28:04.617296 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.617258 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret podName:13da2219-8d3c-4b28-b6b0-38b51415de22 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:20.617242391 +0000 UTC m=+33.455748260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret") pod "global-pull-secret-syncer-r8qkh" (UID: "13da2219-8d3c-4b28-b6b0-38b51415de22") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:28:04.694011 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.693978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:04.694193 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.694020 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:04.694193 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.694105 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:04.694293 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.694180 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:04.694293 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.694245 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:04.694374 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.694338 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:04.718258 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.718213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:04.718433 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.718374 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:28:04.718433 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.718404 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:28:04.718433 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.718418 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fjqwf for pod openshift-network-diagnostics/network-check-target-ssjs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:28:04.718601 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:04.718499 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf podName:c967790c-1a44-4339-aa6e-8becad9f4474 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:20.71846011 +0000 UTC m=+33.556965977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqwf" (UniqueName: "kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf") pod "network-check-target-ssjs9" (UID: "c967790c-1a44-4339-aa6e-8becad9f4474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:28:04.861172 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.861127 2573 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="c6008217d8979d8ef43be30d8d74635a59fb7bc00c1d2cff6a19dab7dc41837b" exitCode=0 Nov 02 19:28:04.861402 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:04.861191 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" event={"ID":"9d9559a8-3f33-4ea2-b478-8b9474d99ed7","Type":"ContainerDied","Data":"c6008217d8979d8ef43be30d8d74635a59fb7bc00c1d2cff6a19dab7dc41837b"} Nov 02 19:28:06.457369 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.457329 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:06.617832 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.617800 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:06.694160 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.694001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:06.694160 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.694001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:06.694160 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:06.694117 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:06.694416 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.694007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:06.694416 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:06.694236 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:06.694416 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:06.694315 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:06.870523 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.870409 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" event={"ID":"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f","Type":"ContainerStarted","Data":"781dd147e5435e030232049bd4cb31a5e60b0b7ea5cc944a6d70ae3441847ed2"} Nov 02 19:28:06.870523 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.870515 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" event={"ID":"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f","Type":"ContainerStarted","Data":"b4a5bcd8424dba8aed40cdaaeca140321a9499caa3c12826a1a7f986225001f5"} Nov 02 19:28:06.870711 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.870554 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" event={"ID":"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f","Type":"ContainerStarted","Data":"989612e6a5f1158474c6dec441bcb524f97818c6ea4f487acbdf250a2395db7b"} Nov 02 19:28:06.870711 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.870591 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" event={"ID":"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f","Type":"ContainerStarted","Data":"4a4b50e620c075f79eb71a58cb121f723c82997799e91efd9556bd0f32a1568c"} Nov 02 19:28:06.870711 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.870618 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" event={"ID":"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f","Type":"ContainerStarted","Data":"9d6be463f2ad2264f621d50af8dc2f29cdfe08488b4abd02729eaebc6ac1c0bb"} Nov 02 19:28:06.870711 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:06.870644 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" event={"ID":"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f","Type":"ContainerStarted","Data":"ee5cd9b9292bdb5fbac662f6a53278e70c70c0778f4195f8fec67d5ce7011242"} Nov 02 19:28:07.875229 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:07.875195 2573 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="e87365c15027646a6c6f0687a0e77bc27983042c9f1ca52e8af0157bca515901" exitCode=0 Nov 02 19:28:07.875759 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:07.875239 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" event={"ID":"9d9559a8-3f33-4ea2-b478-8b9474d99ed7","Type":"ContainerDied","Data":"e87365c15027646a6c6f0687a0e77bc27983042c9f1ca52e8af0157bca515901"} Nov 02 19:28:08.693377 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:08.693344 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:08.693578 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:08.693463 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:08.693578 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:08.693496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:08.693578 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:08.693500 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:08.693731 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:08.693590 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:08.693731 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:08.693700 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:08.881052 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:08.881017 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" event={"ID":"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f","Type":"ContainerStarted","Data":"5b0717e5f58d7069d7af3e2bec1715abd52bb5735a3d7fe9c2aed43e338f50f7"} Nov 02 19:28:08.883843 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:08.883817 2573 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="584262aa807d770a8c6b6a74558d77aef8fd22575d45d7ec1682250fe44018b4" exitCode=0 Nov 02 19:28:08.883954 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:08.883848 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" event={"ID":"9d9559a8-3f33-4ea2-b478-8b9474d99ed7","Type":"ContainerDied","Data":"584262aa807d770a8c6b6a74558d77aef8fd22575d45d7ec1682250fe44018b4"} Nov 02 19:28:09.888846 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:09.888814 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" event={"ID":"9d9559a8-3f33-4ea2-b478-8b9474d99ed7","Type":"ContainerStarted","Data":"bd13b6ffd42554a2d99d2dbecc4f26f9ba197c22e95e310c064b3b2ae5472e8c"} Nov 02 19:28:10.098667 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.098615 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cgzbx" podStartSLOduration=8.710837135 podStartE2EDuration="22.098599369s" totalImagesPullingTime="13.387762234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:10.098010754 +0000 UTC m=+22.936516628" watchObservedRunningTime="2026-11-02 19:28:10.098599369 +0000 UTC m=+22.937105244" Nov 02 19:28:10.170609 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.170524 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:10.377619 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.377577 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:10.599073 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.598995 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:10.694026 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.693985 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:10.694169 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.694062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:10.694169 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.694153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:10.694169 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:10.694157 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:10.694294 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:10.694242 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:10.694351 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:10.694333 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:10.764112 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.764065 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:10.895448 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.895371 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" event={"ID":"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f","Type":"ContainerStarted","Data":"7ba94d02c693793407c900af250be13015ec702bd67b959db8b91724b85d5ee7"} Nov 02 19:28:10.921424 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:10.921197 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:11.113607 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:11.113551 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" podStartSLOduration=6.904243088 podStartE2EDuration="23.113537784s" totalImagesPullingTime="16.209294696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:11.065962194 +0000 UTC m=+23.904468069" watchObservedRunningTime="2026-11-02 19:28:11.113537784 +0000 UTC m=+23.952043659" Nov 02 19:28:11.709614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:11.709583 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:11.875870 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:11.875836 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:11.897321 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:11.897294 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:28:11.897745 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:11.897730 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:28:11.897819 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:11.897807 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:28:11.913666 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:11.913641 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:28:11.913815 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:11.913771 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:28:12.694221 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:12.694187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:12.694366 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:12.694244 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:12.694366 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:12.694331 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:12.694366 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:12.694336 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:12.694508 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:12.694396 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:12.694508 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:12.694490 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:12.898967 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:12.898939 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:28:13.324264 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:13.324226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:13.590001 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:13.589907 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tq4ks"] Nov 02 19:28:13.590172 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:13.590040 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:13.590172 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:13.590156 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:13.594609 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:13.594583 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ssjs9"] Nov 02 19:28:13.594759 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:13.594678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:13.594800 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:13.594755 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:13.595129 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:13.595111 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r8qkh"] Nov 02 19:28:13.595194 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:13.595184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:13.595265 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:13.595251 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:13.900966 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:13.900893 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:28:14.060115 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:14.060085 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:15.285933 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:15.285854 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:15.693566 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:15.693527 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:15.693566 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:15.693531 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:15.693791 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:15.693635 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:15.693791 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:15.693642 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:15.693791 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:15.693708 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:15.693791 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:15.693778 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:15.982732 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:15.982642 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:28:15.982903 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:15.982888 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:28:15.993778 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:15.993737 2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" podUID="84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 19:28:16.002399 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:16.002369 2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" podUID="84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 19:28:16.695067 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:16.695038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:17.506291 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:17.506255 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:17.696564 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:17.696367 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:17.696951 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:17.696416 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:17.696951 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:17.696635 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:17.696951 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:17.696422 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:17.696951 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:17.696701 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:17.696951 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:17.696764 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:17.714247 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:17.714209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:18.596937 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:18.596896 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:18.811875 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:18.811842 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:19.527955 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:19.527925 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:19.693669 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:19.693632 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:19.693669 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:19.693663 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:19.693913 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:19.693754 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:19.693913 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:19.693809 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:19.693913 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:19.693901 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:19.694056 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:19.693954 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:19.818678 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:19.818597 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:19.901614 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:19.901589 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zlvtc_f31b231b-c403-428b-a1a3-f516644ac9e5/dns-node-resolver/0.log" Nov 02 19:28:19.979328 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:19.979288 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:20.202438 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:20.202350 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:20.370681 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:20.370637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:20.502019 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:20.501950 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7bzn9_53fc9047-2f91-4806-acfd-d23b1b0cbc71/node-ca/0.log" Nov 02 19:28:20.544605 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:20.544572 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:20.623728 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:20.623692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:20.623911 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:20.623765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:20.623911 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:20.623840 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:28:20.623911 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:20.623871 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:28:20.623911 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:20.623902 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret podName:13da2219-8d3c-4b28-b6b0-38b51415de22 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:52.623885588 +0000 UTC m=+65.462391454 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret") pod "global-pull-secret-syncer-r8qkh" (UID: "13da2219-8d3c-4b28-b6b0-38b51415de22") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:28:20.623911 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:20.623917 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs podName:19ec87c5-1d05-4829-9a6d-cddcd0f84607 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:52.62391013 +0000 UTC m=+65.462415983 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs") pod "network-metrics-daemon-tq4ks" (UID: "19ec87c5-1d05-4829-9a6d-cddcd0f84607") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:28:20.717607 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:20.717577 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:20.724359 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:20.724335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:20.724482 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:20.724456 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:28:20.724523 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:20.724469 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:28:20.724523 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:20.724492 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fjqwf for pod openshift-network-diagnostics/network-check-target-ssjs9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:28:20.724591 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:20.724536 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf podName:c967790c-1a44-4339-aa6e-8becad9f4474 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:52.72452473 +0000 UTC m=+65.563030596 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjqwf" (UniqueName: "kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf") pod "network-check-target-ssjs9" (UID: "c967790c-1a44-4339-aa6e-8becad9f4474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:28:21.696817 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:21.696788 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:21.697221 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:21.696883 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:21.697221 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:21.696954 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:21.697221 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:21.697041 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:21.697221 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:21.697079 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:21.697221 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:21.697153 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:21.776904 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:21.776871 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:22.010930 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:22.010854 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:22.272259 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:22.272164 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:22.446691 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:22.446660 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:22.619125 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:22.619034 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:23.693869 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:23.693834 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:23.694305 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:23.693899 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:23.694305 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:23.693904 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:23.694305 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:23.693992 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:23.694305 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:23.694061 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:23.694305 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:23.694114 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:23.771070 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:23.771029 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:23.967124 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:23.967048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:24.125977 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:24.125946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:24.282879 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:24.282793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:24.429964 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:24.429933 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:24.580753 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:24.580669 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:24.927679 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:24.927594 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:25.693493 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:25.693455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:25.693493 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:25.693497 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:25.693748 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:25.693455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:25.693748 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:25.693584 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:25.693748 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:25.693661 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:25.693748 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:25.693718 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:25.761033 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:25.760997 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:25.959888 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:25.959798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:26.143761 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:26.143726 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:26.313539 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:26.313440 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:26.477107 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:26.477072 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:26.639886 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:26.639793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:27.694420 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:27.694390 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:27.694971 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:27.694444 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:27.694971 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:27.694553 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:27.694971 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:27.694579 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:27.694971 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:27.694660 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:27.694971 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:27.694735 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:27.763711 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:27.763671 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:27.966399 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:27.966310 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:28.164262 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:28.164232 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:28.364637 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:28.364556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:28.565223 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:28.565182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:29.578344 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:29.578308 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:29.693852 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:29.693822 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:29.694039 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:29.693823 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:29.694039 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:29.693822 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:29.694039 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:29.693952 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:29.694039 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:29.694017 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:29.694251 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:29.694072 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:29.784735 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:29.784694 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:29.959912 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:29.959819 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:30.174726 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:30.174693 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:31.694414 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:31.694378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:31.694912 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:31.694386 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:31.694912 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:31.694523 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:31.694912 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:31.694533 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:31.694912 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:31.694655 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:31.694912 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:31.694713 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:32.491578 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:32.491547 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:32.641052 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:32.641021 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:33.693581 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:33.693548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:33.694021 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:33.693583 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:33.694021 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:33.693548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:33.694021 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:33.693659 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:33.694021 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:33.693767 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:33.694021 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:33.693851 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:34.960315 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:34.960283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:35.164649 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:35.164611 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:35.423325 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:35.423295 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:35.574283 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:35.574248 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:35.693743 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:35.693655 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:35.693888 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:35.693671 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:35.693888 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:35.693767 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tq4ks" podUID="19ec87c5-1d05-4829-9a6d-cddcd0f84607" Nov 02 19:28:35.693888 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:35.693791 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:35.693981 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:35.693873 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r8qkh" podUID="13da2219-8d3c-4b28-b6b0-38b51415de22" Nov 02 19:28:35.693981 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:35.693952 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssjs9" podUID="c967790c-1a44-4339-aa6e-8becad9f4474" Nov 02 19:28:35.791890 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:35.791851 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:35.955113 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:35.955042 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeReady" Nov 02 19:28:35.955252 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:35.955184 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 19:28:36.111733 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.111699 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dbcdb"] Nov 02 19:28:36.113676 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.113660 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-knrmt"] Nov 02 19:28:36.113794 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.113779 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.115502 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.115468 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:28:36.139737 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.139708 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vngml"] Nov 02 19:28:36.141863 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.141844 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.143318 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.143299 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbcdb"] Nov 02 19:28:36.146676 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.146655 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-knrmt"] Nov 02 19:28:36.164060 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.164036 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:28:36.164769 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.164747 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:28:36.164892 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.164775 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dc6gd\"" Nov 02 19:28:36.164892 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.164797 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:28:36.165029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.164949 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:28:36.165029 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.164984 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fzxnx\"" Nov 02 19:28:36.169198 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.169170 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vngml"] Nov 02 19:28:36.224611 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224523 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/405d3ed6-8332-46f0-b97d-326543807323-data-volume\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.224611 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54czs\" (UniqueName: \"kubernetes.io/projected/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-kube-api-access-54czs\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.224818 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224652 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6j95\" (UniqueName: \"kubernetes.io/projected/405d3ed6-8332-46f0-b97d-326543807323-kube-api-access-q6j95\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.224818 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224696 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7acda1fa-9e95-4357-ba7b-7315edb4352e-cert\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:28:36.224818 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/405d3ed6-8332-46f0-b97d-326543807323-crio-socket\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.224818 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/405d3ed6-8332-46f0-b97d-326543807323-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.224993 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-metrics-tls\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.224993 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcgb7\" (UniqueName: \"kubernetes.io/projected/7acda1fa-9e95-4357-ba7b-7315edb4352e-kube-api-access-qcgb7\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:28:36.224993 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-config-volume\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.224993 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-tmp-dir\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.224993 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.224965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/405d3ed6-8332-46f0-b97d-326543807323-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.264516 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.264465 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:28:36.264687 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.264600 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:28:36.264753 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.264738 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k9tfs\"" Nov 02 19:28:36.264854 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.264814 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:28:36.264919 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.264904 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:28:36.264979 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.264956 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:28:36.326067 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-config-volume\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.326170 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326084 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-tmp-dir\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.326170 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/405d3ed6-8332-46f0-b97d-326543807323-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.326170 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/405d3ed6-8332-46f0-b97d-326543807323-data-volume\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.326170 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326164 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54czs\" (UniqueName: \"kubernetes.io/projected/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-kube-api-access-54czs\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.326353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6j95\" (UniqueName: \"kubernetes.io/projected/405d3ed6-8332-46f0-b97d-326543807323-kube-api-access-q6j95\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.326353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7acda1fa-9e95-4357-ba7b-7315edb4352e-cert\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:28:36.326353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/405d3ed6-8332-46f0-b97d-326543807323-crio-socket\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.326521 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326428 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/405d3ed6-8332-46f0-b97d-326543807323-crio-socket\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.326597 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326543 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-tmp-dir\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.326597 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326576 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/405d3ed6-8332-46f0-b97d-326543807323-data-volume\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.326597 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/405d3ed6-8332-46f0-b97d-326543807323-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.326760 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-metrics-tls\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.326760 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcgb7\" (UniqueName: \"kubernetes.io/projected/7acda1fa-9e95-4357-ba7b-7315edb4352e-kube-api-access-qcgb7\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:28:36.326915 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.326896 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-config-volume\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.327170 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.327144 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/405d3ed6-8332-46f0-b97d-326543807323-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.330076 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.330046 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/405d3ed6-8332-46f0-b97d-326543807323-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.330180 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.330048 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-metrics-tls\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.330249 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.330182 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7acda1fa-9e95-4357-ba7b-7315edb4352e-cert\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:28:36.381114 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.381087 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcgb7\" (UniqueName: \"kubernetes.io/projected/7acda1fa-9e95-4357-ba7b-7315edb4352e-kube-api-access-qcgb7\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:28:36.381765 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.381742 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6j95\" (UniqueName: \"kubernetes.io/projected/405d3ed6-8332-46f0-b97d-326543807323-kube-api-access-q6j95\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.381999 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.381983 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54czs\" (UniqueName: \"kubernetes.io/projected/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-kube-api-access-54czs\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.427014 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.426956 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:36.429736 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.429711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:28:36.455579 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.455540 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:28:36.463015 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.462866 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:36.508021 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.507926 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t"] Nov 02 19:28:36.510833 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.510797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:28:36.562107 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.562074 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k9469\"" Nov 02 19:28:36.562107 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.562099 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 19:28:36.567824 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.567794 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t"] Nov 02 19:28:36.628303 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.628265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2de0d2d-3838-4979-b2ec-c3ed7df1beb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mkf7t\" (UID: \"a2de0d2d-3838-4979-b2ec-c3ed7df1beb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:28:36.729234 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.729200 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2de0d2d-3838-4979-b2ec-c3ed7df1beb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mkf7t\" (UID: \"a2de0d2d-3838-4979-b2ec-c3ed7df1beb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:28:36.731533 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.731511 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2de0d2d-3838-4979-b2ec-c3ed7df1beb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mkf7t\" (UID: \"a2de0d2d-3838-4979-b2ec-c3ed7df1beb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:28:36.752519 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.752414 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:36.836174 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.836134 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:28:36.839048 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.838911 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbcdb"] Nov 02 19:28:36.843687 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:36.843661 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbe0bd1_bc34_4664_9fa5_5d13952fe3d2.slice/crio-488c6e0df5e686add29118b2f232ca1d1e0f5e218f2361ce793f0b0934bde8b2 WatchSource:0}: Error finding container 488c6e0df5e686add29118b2f232ca1d1e0f5e218f2361ce793f0b0934bde8b2: Status 404 returned error can't find the container with id 488c6e0df5e686add29118b2f232ca1d1e0f5e218f2361ce793f0b0934bde8b2 Nov 02 19:28:36.857246 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.857180 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-knrmt"] Nov 02 19:28:36.860503 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:36.860449 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acda1fa_9e95_4357_ba7b_7315edb4352e.slice/crio-436a8b4fe9ec3b4600bfa808f8f091d66e6f712709acf0841b833d5c3d55fc79 WatchSource:0}: Error finding container 436a8b4fe9ec3b4600bfa808f8f091d66e6f712709acf0841b833d5c3d55fc79: Status 404 returned error can't find the container with id 436a8b4fe9ec3b4600bfa808f8f091d66e6f712709acf0841b833d5c3d55fc79 Nov 02 19:28:36.861502 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.861326 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vngml"] Nov 02 19:28:36.866635 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:36.866605 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405d3ed6_8332_46f0_b97d_326543807323.slice/crio-bfc75af335b31db6f21ecab7e09792756a3e511287d837b8275c12a1c04788d3 WatchSource:0}: Error finding container bfc75af335b31db6f21ecab7e09792756a3e511287d837b8275c12a1c04788d3: Status 404 returned error can't find the container with id bfc75af335b31db6f21ecab7e09792756a3e511287d837b8275c12a1c04788d3 Nov 02 19:28:36.954623 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.954583 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-knrmt" event={"ID":"7acda1fa-9e95-4357-ba7b-7315edb4352e","Type":"ContainerStarted","Data":"436a8b4fe9ec3b4600bfa808f8f091d66e6f712709acf0841b833d5c3d55fc79"} Nov 02 19:28:36.955690 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.955657 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbcdb" event={"ID":"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2","Type":"ContainerStarted","Data":"488c6e0df5e686add29118b2f232ca1d1e0f5e218f2361ce793f0b0934bde8b2"} Nov 02 19:28:36.957074 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.957053 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vngml" event={"ID":"405d3ed6-8332-46f0-b97d-326543807323","Type":"ContainerStarted","Data":"eb884b8ce085757be64e023c75c8fbf83cfc2a6888b5a2a73607060dbbad8b78"} Nov 02 19:28:36.957177 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:36.957095 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vngml" event={"ID":"405d3ed6-8332-46f0-b97d-326543807323","Type":"ContainerStarted","Data":"bfc75af335b31db6f21ecab7e09792756a3e511287d837b8275c12a1c04788d3"} Nov 02 19:28:37.200151 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.200119 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t"] Nov 02 19:28:37.203221 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:37.203194 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2de0d2d_3838_4979_b2ec_c3ed7df1beb9.slice/crio-888426523d429d534a23b252e334071e1313754d18713d8f3dbaf7658ddee31c WatchSource:0}: Error finding container 888426523d429d534a23b252e334071e1313754d18713d8f3dbaf7658ddee31c: Status 404 returned error can't find the container with id 888426523d429d534a23b252e334071e1313754d18713d8f3dbaf7658ddee31c Nov 02 19:28:37.697435 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.697410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:37.697622 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.697537 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:37.697824 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.697807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:37.747081 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.747053 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:28:37.747277 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.747168 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:28:37.750572 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.747552 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n2mjf\"" Nov 02 19:28:37.750572 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.747953 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-c8nwr\"" Nov 02 19:28:37.750572 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.748008 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbvgr\"" Nov 02 19:28:37.750572 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.748218 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:28:37.750572 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.748281 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:28:37.961108 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:37.961036 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" event={"ID":"a2de0d2d-3838-4979-b2ec-c3ed7df1beb9","Type":"ContainerStarted","Data":"888426523d429d534a23b252e334071e1313754d18713d8f3dbaf7658ddee31c"} Nov 02 19:28:38.546281 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:38.546220 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:38.966018 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:38.965975 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" event={"ID":"a2de0d2d-3838-4979-b2ec-c3ed7df1beb9","Type":"ContainerStarted","Data":"d92adb4a8335b25e607faeabad3745bfc42663c87fe4c0e27d0b66f5dde1f6c6"} Nov 02 19:28:38.966248 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:38.966227 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:28:38.968764 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:38.968694 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vngml" event={"ID":"405d3ed6-8332-46f0-b97d-326543807323","Type":"ContainerStarted","Data":"2a6eb42b790ba7841af64325ca95d152ccd46f2c68e2193692f00cc3e92119f4"} Nov 02 19:28:38.972670 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:38.972652 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:28:39.073281 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:39.073164 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" podStartSLOduration=1.454082782 podStartE2EDuration="3.073150308s" totalImagesPullingTime="1.619067526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:39.072851589 +0000 UTC m=+51.911357466" watchObservedRunningTime="2026-11-02 19:28:39.073150308 +0000 UTC m=+51.911656184" Nov 02 19:28:39.714662 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:39.714623 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:39.917729 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:39.917690 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:39.974454 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:39.973882 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbcdb" event={"ID":"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2","Type":"ContainerStarted","Data":"418049ed331e37c5c0f27a460767bb665e958977e2d1ed7292efe6122347a455"} Nov 02 19:28:39.974454 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:39.973936 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbcdb" event={"ID":"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2","Type":"ContainerStarted","Data":"eae051b4b298e033b91e4ea8e2c8f1ed597b0224992ca6572a831481ce81ec71"} Nov 02 19:28:40.077196 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.077164 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq"] Nov 02 19:28:40.079787 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.079756 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.080162 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.080129 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:40.085245 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.085166 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dbcdb" podStartSLOduration=1.7712504180000002 podStartE2EDuration="4.08514984s" totalImagesPullingTime="2.313899422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:40.084942219 +0000 UTC m=+52.923448114" watchObservedRunningTime="2026-11-02 19:28:40.08514984 +0000 UTC m=+52.923655716" Nov 02 19:28:40.130373 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.130338 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:28:40.130567 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.130432 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 19:28:40.130720 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.130697 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:28:40.130845 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.130817 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fqq6c\"" Nov 02 19:28:40.130902 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.130859 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:28:40.130955 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.130934 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 19:28:40.139660 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.139628 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq"] Nov 02 19:28:40.157909 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.157874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8kl\" (UniqueName: \"kubernetes.io/projected/2867a688-7f22-4419-a69c-aef174888edc-kube-api-access-dl8kl\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.158074 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.157941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.158137 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.158082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2867a688-7f22-4419-a69c-aef174888edc-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.158137 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.158122 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.258861 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.258830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.258861 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.258866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8kl\" (UniqueName: \"kubernetes.io/projected/2867a688-7f22-4419-a69c-aef174888edc-kube-api-access-dl8kl\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.259071 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.258906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.259071 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.258955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2867a688-7f22-4419-a69c-aef174888edc-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.259071 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:40.258960 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Nov 02 19:28:40.259071 ip-10-0-131-244 kubenswrapper[2573]: E1102 19:28:40.259023 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-tls podName:2867a688-7f22-4419-a69c-aef174888edc nodeName:}" failed. No retries permitted until 2026-11-02 19:28:40.759008085 +0000 UTC m=+53.597513941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-tls") pod "prometheus-operator-9cd89b74c-4rpfq" (UID: "2867a688-7f22-4419-a69c-aef174888edc") : secret "prometheus-operator-tls" not found Nov 02 19:28:40.259656 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.259637 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2867a688-7f22-4419-a69c-aef174888edc-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.261104 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.261082 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.315879 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.315847 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8kl\" (UniqueName: \"kubernetes.io/projected/2867a688-7f22-4419-a69c-aef174888edc-kube-api-access-dl8kl\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.518172 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.518079 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:40.762820 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.762779 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.765144 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.765123 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:40.978600 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.978563 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-knrmt" event={"ID":"7acda1fa-9e95-4357-ba7b-7315edb4352e","Type":"ContainerStarted","Data":"16868d148b644d51641f128c9d83c37f2d37a62f0d233f6ab832213152a7be3c"} Nov 02 19:28:40.980706 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.980680 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vngml" event={"ID":"405d3ed6-8332-46f0-b97d-326543807323","Type":"ContainerStarted","Data":"edc4c0ba9d843f85bbed24a3de68b24567c04ae855e24bef8e00c9ec3117a813"} Nov 02 19:28:40.980837 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.980734 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:40.995506 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:40.995460 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:28:41.087245 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:41.087189 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-knrmt" podStartSLOduration=1.7574811540000002 podStartE2EDuration="5.087174821s" totalImagesPullingTime="3.329693667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:41.086301141 +0000 UTC m=+53.924807034" watchObservedRunningTime="2026-11-02 19:28:41.087174821 +0000 UTC m=+53.925680705" Nov 02 19:28:41.197083 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:41.197018 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vngml" podStartSLOduration=1.971685134 podStartE2EDuration="5.197000974s" totalImagesPullingTime="3.22531584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:28:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:41.196990854 +0000 UTC m=+54.035496728" watchObservedRunningTime="2026-11-02 19:28:41.197000974 +0000 UTC m=+54.035506852" Nov 02 19:28:41.356695 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:41.356620 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq"] Nov 02 19:28:41.360260 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:41.360231 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2867a688_7f22_4419_a69c_aef174888edc.slice/crio-770569fc3162f6a8fa8c1dc5e649325d3108c12182d41839520c33c02428e386 WatchSource:0}: Error finding container 770569fc3162f6a8fa8c1dc5e649325d3108c12182d41839520c33c02428e386: Status 404 returned error can't find the container with id 770569fc3162f6a8fa8c1dc5e649325d3108c12182d41839520c33c02428e386 Nov 02 19:28:41.985381 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:41.985347 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" event={"ID":"2867a688-7f22-4419-a69c-aef174888edc","Type":"ContainerStarted","Data":"770569fc3162f6a8fa8c1dc5e649325d3108c12182d41839520c33c02428e386"} Nov 02 19:28:42.463464 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:42.463428 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:43.992204 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:43.992170 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" event={"ID":"2867a688-7f22-4419-a69c-aef174888edc","Type":"ContainerStarted","Data":"805a7b1d615a4978a0674f845c9519ec7508c2ab44e1d24045a99339da93ec30"} Nov 02 19:28:43.992590 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:43.992222 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" event={"ID":"2867a688-7f22-4419-a69c-aef174888edc","Type":"ContainerStarted","Data":"22e405650e699b557c7a4fcfdb46ee8ee00e0f06abb14293ed3e6bdfffb8d305"} Nov 02 19:28:44.101847 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:44.101799 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" podStartSLOduration=2.476860344 podStartE2EDuration="4.101785939s" totalImagesPullingTime="1.624925595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:44.101556654 +0000 UTC m=+56.940062528" watchObservedRunningTime="2026-11-02 19:28:44.101785939 +0000 UTC m=+56.940291819" Nov 02 19:28:45.532730 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:45.532695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:46.004307 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:46.004273 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:28:46.919417 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:46.919377 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:47.133349 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.133309 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:47.312776 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.312692 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:47.482841 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.482810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:47.535729 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.535693 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw"] Nov 02 19:28:47.538067 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.538050 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.583459 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.583364 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cswh5"] Nov 02 19:28:47.586147 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.586121 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.597324 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.597300 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c6zv8\"" Nov 02 19:28:47.597324 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.597315 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:28:47.630714 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.630677 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw"] Nov 02 19:28:47.635592 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.635562 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 19:28:47.639920 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.639900 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 19:28:47.662847 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.662813 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:47.697619 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.697597 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:28:47.697791 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.697690 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njtg7\"" Nov 02 19:28:47.697836 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.697822 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:28:47.717563 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-wtmp\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.717563 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717563 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-metrics-client-ca\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.717740 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.717740 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.717801 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-textfile\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.717801 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717790 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.717861 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-root\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.717861 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717828 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.717919 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/43a8f63a-57a0-4be5-b55f-621b67342950-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.717953 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgsrp\" (UniqueName: \"kubernetes.io/projected/43a8f63a-57a0-4be5-b55f-621b67342950-kube-api-access-bgsrp\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.717984 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.717959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-sys\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.718028 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.718013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.718062 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.718039 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.718102 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.718090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-tls\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.718137 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.718116 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6828\" (UniqueName: \"kubernetes.io/projected/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-kube-api-access-r6828\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.724994 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.724974 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:28:47.819250 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/43a8f63a-57a0-4be5-b55f-621b67342950-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.819433 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsrp\" (UniqueName: \"kubernetes.io/projected/43a8f63a-57a0-4be5-b55f-621b67342950-kube-api-access-bgsrp\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.819433 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-sys\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.819433 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.819433 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.819433 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-tls\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.819433 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6828\" (UniqueName: \"kubernetes.io/projected/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-kube-api-access-r6828\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.819433 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819402 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-sys\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.819834 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-wtmp\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.819834 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-metrics-client-ca\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.819834 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.819834 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.819834 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-textfile\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.819834 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.819834 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-root\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.819834 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819680 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/43a8f63a-57a0-4be5-b55f-621b67342950-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.819834 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.820266 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.819885 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-wtmp\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.820266 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.820158 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-textfile\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.820266 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.820169 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-metrics-client-ca\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.820266 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.820192 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.820266 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.820254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-root\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.820871 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.820848 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.820932 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.820910 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.822085 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.822063 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.822250 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.822203 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.822353 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.822336 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-tls\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.822673 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.822650 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.822744 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.822647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:47.873551 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.873521 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6828\" (UniqueName: \"kubernetes.io/projected/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-kube-api-access-r6828\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.873685 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.873629 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgsrp\" (UniqueName: \"kubernetes.io/projected/43a8f63a-57a0-4be5-b55f-621b67342950-kube-api-access-bgsrp\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:47.900514 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.900460 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:28:47.911756 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:47.911727 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf0d624_3b0c_4ceb_897a_c9d8b54213e2.slice/crio-48fe43dd697504c22f3b846239e421a443aac2854ece9de26c4c8661810f20e6 WatchSource:0}: Error finding container 48fe43dd697504c22f3b846239e421a443aac2854ece9de26c4c8661810f20e6: Status 404 returned error can't find the container with id 48fe43dd697504c22f3b846239e421a443aac2854ece9de26c4c8661810f20e6 Nov 02 19:28:47.978875 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:47.978844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:48.006208 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:48.006180 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cswh5" event={"ID":"daf0d624-3b0c-4ceb-897a-c9d8b54213e2","Type":"ContainerStarted","Data":"48fe43dd697504c22f3b846239e421a443aac2854ece9de26c4c8661810f20e6"} Nov 02 19:28:48.151240 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:48.151152 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:28:48.173599 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:48.173566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:48.561133 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:48.561038 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw"] Nov 02 19:28:48.601247 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:48.601215 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a8f63a_57a0_4be5_b55f_621b67342950.slice/crio-0f5bd0153977ea80cbdeebb54fbaafe9846cf48385a69ea2c02d3dd89d04e4ec WatchSource:0}: Error finding container 0f5bd0153977ea80cbdeebb54fbaafe9846cf48385a69ea2c02d3dd89d04e4ec: Status 404 returned error can't find the container with id 0f5bd0153977ea80cbdeebb54fbaafe9846cf48385a69ea2c02d3dd89d04e4ec Nov 02 19:28:48.859438 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:48.859343 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:49.010617 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.010579 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" event={"ID":"43a8f63a-57a0-4be5-b55f-621b67342950","Type":"ContainerStarted","Data":"0f5bd0153977ea80cbdeebb54fbaafe9846cf48385a69ea2c02d3dd89d04e4ec"} Nov 02 19:28:49.128931 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.128888 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:49.711532 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.711495 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:49.759068 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.759036 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:28:49.761798 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.761775 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.811231 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.811170 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-snhdz\"" Nov 02 19:28:49.860758 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.860717 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:28:49.918748 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.918713 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 19:28:49.918959 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.918769 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 19:28:49.918959 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.918717 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 19:28:49.918959 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.918810 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 19:28:49.918959 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.918713 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 19:28:49.918959 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.918851 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 19:28:49.918959 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.918862 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 19:28:49.919201 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.918981 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 19:28:49.919201 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.919029 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 19:28:49.937800 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.937764 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.937992 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.937822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzm9\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.937992 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.937861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.937992 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.937887 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.937992 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.937952 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.938188 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.938019 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.938188 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.938048 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.938188 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.938108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.938188 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.938131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.938188 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.938166 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.938412 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.938273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.938412 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.938314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:49.938412 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:49.938341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.021885 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.021791 2573 generic.go:356] "Generic (PLEG): container finished" podID="daf0d624-3b0c-4ceb-897a-c9d8b54213e2" containerID="939f95f1a375f7a9abf3dbb0025e4195d45dca7c2a0e6d1bffea6632f88ed1b0" exitCode=0 Nov 02 19:28:50.021885 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.021851 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cswh5" event={"ID":"daf0d624-3b0c-4ceb-897a-c9d8b54213e2","Type":"ContainerDied","Data":"939f95f1a375f7a9abf3dbb0025e4195d45dca7c2a0e6d1bffea6632f88ed1b0"} Nov 02 19:28:50.039274 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039391 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039391 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039326 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039391 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039567 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039567 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039567 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039567 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039567 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039794 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzm9\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039794 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039637 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039794 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.039794 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.039695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.040392 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.040275 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.041564 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.040613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.041564 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.041247 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.042900 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.042873 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.043147 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.042938 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.044082 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.044032 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.044082 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.044043 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.044892 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.044668 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.044892 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.044854 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.045687 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.045661 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.046319 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.046294 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.046414 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.046319 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.097246 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.097215 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzm9\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.104358 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.104329 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-64c99bbd67-6qj9p"] Nov 02 19:28:50.107500 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.107461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.157828 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.157797 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 19:28:50.158013 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.157887 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 19:28:50.158013 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.157902 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 19:28:50.158013 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.157900 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 19:28:50.158013 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.157911 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 19:28:50.158013 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.157966 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-52vlcnle3t3js\"" Nov 02 19:28:50.158175 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.158042 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4x665\"" Nov 02 19:28:50.167908 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.167878 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64c99bbd67-6qj9p"] Nov 02 19:28:50.242094 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.242040 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.242219 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.242149 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhrj\" (UniqueName: \"kubernetes.io/projected/de1657c9-1cb9-44bb-b505-adf726ab2540-kube-api-access-5nhrj\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.242219 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.242178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.242286 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.242221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.242286 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.242243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-grpc-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.242286 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.242273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de1657c9-1cb9-44bb-b505-adf726ab2540-tmp\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.242392 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.242370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.242426 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.242404 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.242508 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.242495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1657c9-1cb9-44bb-b505-adf726ab2540-metrics-client-ca\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.343495 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.343446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.343579 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.343508 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.343579 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.343567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1657c9-1cb9-44bb-b505-adf726ab2540-metrics-client-ca\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.343678 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.343632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.343678 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.343672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhrj\" (UniqueName: \"kubernetes.io/projected/de1657c9-1cb9-44bb-b505-adf726ab2540-kube-api-access-5nhrj\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.343782 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.343699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.343782 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.343726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.343782 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.343750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-grpc-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.343923 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.343793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de1657c9-1cb9-44bb-b505-adf726ab2540-tmp\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.344429 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.344195 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de1657c9-1cb9-44bb-b505-adf726ab2540-tmp\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.344429 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.344383 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1657c9-1cb9-44bb-b505-adf726ab2540-metrics-client-ca\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.346572 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.346542 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.346744 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.346717 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.346930 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.346904 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.346995 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.346962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.347180 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.347154 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-grpc-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.347829 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.347810 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.383074 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.382894 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:28:50.400442 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.400413 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhrj\" (UniqueName: \"kubernetes.io/projected/de1657c9-1cb9-44bb-b505-adf726ab2540-kube-api-access-5nhrj\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.427739 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.427702 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:50.542170 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.542049 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:50.707928 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.707896 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:50.855936 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.855721 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64c99bbd67-6qj9p"] Nov 02 19:28:50.858727 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:50.858698 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1657c9_1cb9_44bb_b505_adf726ab2540.slice/crio-6446e2aebac2a76a71da81fe3d07269e7e2b1f4deef71b0608e84f70b5c6f130 WatchSource:0}: Error finding container 6446e2aebac2a76a71da81fe3d07269e7e2b1f4deef71b0608e84f70b5c6f130: Status 404 returned error can't find the container with id 6446e2aebac2a76a71da81fe3d07269e7e2b1f4deef71b0608e84f70b5c6f130 Nov 02 19:28:50.909025 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.908879 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:28:50.911968 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:50.911933 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4c68e9_066f_4de3_9657_27fa1831f327.slice/crio-8dfcdfe17db9bc3ac0a2af09c527fb7c32bdf8b17eb49aad1719f1e8638a3a8a WatchSource:0}: Error finding container 8dfcdfe17db9bc3ac0a2af09c527fb7c32bdf8b17eb49aad1719f1e8638a3a8a: Status 404 returned error can't find the container with id 8dfcdfe17db9bc3ac0a2af09c527fb7c32bdf8b17eb49aad1719f1e8638a3a8a Nov 02 19:28:50.987956 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:50.987922 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbcdb" Nov 02 19:28:51.026553 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.026519 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" event={"ID":"de1657c9-1cb9-44bb-b505-adf726ab2540","Type":"ContainerStarted","Data":"6446e2aebac2a76a71da81fe3d07269e7e2b1f4deef71b0608e84f70b5c6f130"} Nov 02 19:28:51.027948 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.027923 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerStarted","Data":"8dfcdfe17db9bc3ac0a2af09c527fb7c32bdf8b17eb49aad1719f1e8638a3a8a"} Nov 02 19:28:51.030698 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.030670 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cswh5" event={"ID":"daf0d624-3b0c-4ceb-897a-c9d8b54213e2","Type":"ContainerStarted","Data":"4729948261cea84b11f2cb03ba51b8171360aa30ab7f84d2fbbc207c1856da7b"} Nov 02 19:28:51.030794 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.030724 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cswh5" event={"ID":"daf0d624-3b0c-4ceb-897a-c9d8b54213e2","Type":"ContainerStarted","Data":"9ba9f975bcf82dd9da8bbfa4f54526d4f3b1ad2da2a1d6ff66fa4084d862a57d"} Nov 02 19:28:51.032721 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.032697 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" event={"ID":"43a8f63a-57a0-4be5-b55f-621b67342950","Type":"ContainerStarted","Data":"8be6b51386e0945826e5370fd3031b5d8e482bec1973ab88442af9b93fbdbc63"} Nov 02 19:28:51.032818 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.032747 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" event={"ID":"43a8f63a-57a0-4be5-b55f-621b67342950","Type":"ContainerStarted","Data":"b2e4268a8ed434db250dfa64afb9963292209a340c8b8bd0bbb01c8e133c7afa"} Nov 02 19:28:51.032818 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.032767 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" event={"ID":"43a8f63a-57a0-4be5-b55f-621b67342950","Type":"ContainerStarted","Data":"403e387da8f8b6e6377be6802b4eb0d9d9b16dc97c4ff5bdba8b943619e145d7"} Nov 02 19:28:51.219991 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.216857 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cswh5" podStartSLOduration=2.8327675 podStartE2EDuration="4.216840719s" totalImagesPullingTime="1.315979879s" totalInitContainerRuntime="68.09334ms" isStatefulPod=true podCreationTimestamp="2026-11-02 19:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:51.215510887 +0000 UTC m=+64.054016762" watchObservedRunningTime="2026-11-02 19:28:51.216840719 +0000 UTC m=+64.055346593" Nov 02 19:28:51.246683 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.246646 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:51.328015 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.327957 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" podStartSLOduration=2.70937927 podStartE2EDuration="4.327943817s" totalImagesPullingTime="1.618564547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:51.327522446 +0000 UTC m=+64.166028320" watchObservedRunningTime="2026-11-02 19:28:51.327943817 +0000 UTC m=+64.166449691" Nov 02 19:28:51.470586 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.470501 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:51.634973 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.634943 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:51.847388 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:51.847295 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:52.025886 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.025841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:52.235933 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.235897 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:52.332294 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.332260 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2hv87"] Nov 02 19:28:52.335146 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.335124 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:52.378768 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.378732 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-868bfbd57c-9kws8"] Nov 02 19:28:52.382197 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.382158 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.387402 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.387259 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 19:28:52.387402 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.387378 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-njjjh\"" Nov 02 19:28:52.394776 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.394751 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2hv87"] Nov 02 19:28:52.419264 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.419221 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:52.437136 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.437101 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 19:28:52.437346 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.437169 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 19:28:52.465214 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.465183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/107e58b3-1281-416a-b713-8efc20576e9f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:52.465214 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.465215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvb8m\" (UniqueName: \"kubernetes.io/projected/107e58b3-1281-416a-b713-8efc20576e9f-kube-api-access-dvb8m\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:52.466893 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.466862 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-868bfbd57c-9kws8"] Nov 02 19:28:52.486884 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.486812 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 19:28:52.487111 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.487096 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4jh97\"" Nov 02 19:28:52.487259 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.487239 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8h3sq919c5rg5\"" Nov 02 19:28:52.487382 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.487262 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 19:28:52.566046 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.566004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-tls\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.566233 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.566070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-client-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.566233 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.566193 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.566341 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.566256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/107e58b3-1281-416a-b713-8efc20576e9f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:52.566341 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.566300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvb8m\" (UniqueName: \"kubernetes.io/projected/107e58b3-1281-416a-b713-8efc20576e9f-kube-api-access-dvb8m\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:52.566441 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.566354 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48eccb46-4917-49bb-b8f0-f40cf28de5f6-audit-log\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.566441 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.566378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjvdn\" (UniqueName: \"kubernetes.io/projected/48eccb46-4917-49bb-b8f0-f40cf28de5f6-kube-api-access-kjvdn\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.566441 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.566416 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-client-certs\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.566615 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.566514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-metrics-server-audit-profiles\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.569089 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.569063 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/107e58b3-1281-416a-b713-8efc20576e9f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:52.606515 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.606467 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:52.621507 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.621457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvb8m\" (UniqueName: \"kubernetes.io/projected/107e58b3-1281-416a-b713-8efc20576e9f-kube-api-access-dvb8m\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:52.649117 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.649091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:52.667655 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.667621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.667800 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.667674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:52.667800 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.667726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48eccb46-4917-49bb-b8f0-f40cf28de5f6-audit-log\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.667918 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.667843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvdn\" (UniqueName: \"kubernetes.io/projected/48eccb46-4917-49bb-b8f0-f40cf28de5f6-kube-api-access-kjvdn\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.667918 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.667887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:52.667918 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.667908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-client-certs\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.668200 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.667935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-metrics-server-audit-profiles\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.668200 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.667978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-tls\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.668200 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.668004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-client-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.668200 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.668120 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48eccb46-4917-49bb-b8f0-f40cf28de5f6-audit-log\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.668426 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.668404 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.669219 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.669188 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-metrics-server-audit-profiles\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.670423 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.670404 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-client-certs\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.670569 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.670514 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-tls\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.670619 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.670572 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-client-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.716835 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.716801 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:28:52.716970 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.716894 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:28:52.722056 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.722032 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:52.722263 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.722240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:52.724371 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.724349 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvdn\" (UniqueName: \"kubernetes.io/projected/48eccb46-4917-49bb-b8f0-f40cf28de5f6-kube-api-access-kjvdn\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:52.769157 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.769076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:52.773101 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.773074 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n2mjf\"" Nov 02 19:28:52.774131 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.774110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:28:52.776437 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.776414 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-c8nwr\"" Nov 02 19:28:52.777997 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.777980 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:28:52.818079 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.818047 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:28:52.868989 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.868958 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:28:52.873501 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.873463 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:52.997226 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:52.996978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:28:53.012381 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.012355 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2hv87"] Nov 02 19:28:53.015327 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.015301 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-ckpzv"] Nov 02 19:28:53.017848 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.017825 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:28:53.021571 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:53.021507 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107e58b3_1281_416a_b713_8efc20576e9f.slice/crio-884442753f0436d3d0b3c66bf30628040932f27f63cc660173957f4be66631ae WatchSource:0}: Error finding container 884442753f0436d3d0b3c66bf30628040932f27f63cc660173957f4be66631ae: Status 404 returned error can't find the container with id 884442753f0436d3d0b3c66bf30628040932f27f63cc660173957f4be66631ae Nov 02 19:28:53.043511 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.043393 2573 generic.go:356] "Generic (PLEG): container finished" podID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerID="e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af" exitCode=0 Nov 02 19:28:53.043684 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.043525 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerDied","Data":"e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af"} Nov 02 19:28:53.046277 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.046251 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" event={"ID":"107e58b3-1281-416a-b713-8efc20576e9f","Type":"ContainerStarted","Data":"884442753f0436d3d0b3c66bf30628040932f27f63cc660173957f4be66631ae"} Nov 02 19:28:53.062086 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.060847 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbvgr\"" Nov 02 19:28:53.068189 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.068169 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-4ltnd\"" Nov 02 19:28:53.071710 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.071462 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:28:53.071832 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.071735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkdj\" (UniqueName: \"kubernetes.io/projected/b5ba7edd-403e-423f-b9a6-92c08b7881a9-kube-api-access-7rkdj\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:28:53.071895 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.071842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ba7edd-403e-423f-b9a6-92c08b7881a9-tmp\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:28:53.081100 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.081033 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-ckpzv"] Nov 02 19:28:53.152866 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.152835 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r8qkh"] Nov 02 19:28:53.154746 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.154722 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tq4ks"] Nov 02 19:28:53.154897 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.154883 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 19:28:53.155041 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.155011 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 19:28:53.156987 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:53.156917 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13da2219_8d3c_4b28_b6b0_38b51415de22.slice/crio-c00e59c137866958842ee52e3b2741e2f170e25f718ad32b342a18ab108add41 WatchSource:0}: Error finding container c00e59c137866958842ee52e3b2741e2f170e25f718ad32b342a18ab108add41: Status 404 returned error can't find the container with id c00e59c137866958842ee52e3b2741e2f170e25f718ad32b342a18ab108add41 Nov 02 19:28:53.157498 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:53.157454 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ec87c5_1d05_4829_9a6d_cddcd0f84607.slice/crio-2f54c0ecffa913bcc377365538a402de44e285bd8b48c5aa16707496b8cb3a02 WatchSource:0}: Error finding container 2f54c0ecffa913bcc377365538a402de44e285bd8b48c5aa16707496b8cb3a02: Status 404 returned error can't find the container with id 2f54c0ecffa913bcc377365538a402de44e285bd8b48c5aa16707496b8cb3a02 Nov 02 19:28:53.157821 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.157791 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:53.173124 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.172551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkdj\" (UniqueName: \"kubernetes.io/projected/b5ba7edd-403e-423f-b9a6-92c08b7881a9-kube-api-access-7rkdj\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:28:53.173124 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.172615 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ba7edd-403e-423f-b9a6-92c08b7881a9-tmp\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:28:53.173124 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.173079 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ba7edd-403e-423f-b9a6-92c08b7881a9-tmp\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:28:53.204287 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.204250 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-57ddbd48bb-l5d62"] Nov 02 19:28:53.207959 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.207927 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.227409 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.227377 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkdj\" (UniqueName: \"kubernetes.io/projected/b5ba7edd-403e-423f-b9a6-92c08b7881a9-kube-api-access-7rkdj\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:28:53.258468 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.258409 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 19:28:53.259454 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.259430 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 19:28:53.259588 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.259533 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 19:28:53.259828 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.259803 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 19:28:53.259951 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.259929 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 19:28:53.260249 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.260214 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mz7zj\"" Nov 02 19:28:53.260430 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.260409 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 19:28:53.270574 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.270529 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57ddbd48bb-l5d62"] Nov 02 19:28:53.274022 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.273960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsz7l\" (UniqueName: \"kubernetes.io/projected/e0180ace-54a9-4796-a2ab-f069f965573c-kube-api-access-zsz7l\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.274022 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.274003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.274157 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.274044 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-metrics-client-ca\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.274157 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.274059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.274157 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.274082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.274157 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.274127 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-federate-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.274270 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.274163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-serving-certs-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.274270 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.274187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.332886 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.332849 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:28:53.372742 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.372705 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-868bfbd57c-9kws8"] Nov 02 19:28:53.374837 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.374812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-federate-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.374928 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.374867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-serving-certs-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.374928 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.374900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.375043 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.374935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsz7l\" (UniqueName: \"kubernetes.io/projected/e0180ace-54a9-4796-a2ab-f069f965573c-kube-api-access-zsz7l\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.375043 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.374965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.375043 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.375015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-metrics-client-ca\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.375175 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.375041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.375175 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.375077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.376090 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.376061 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-serving-certs-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.376186 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.376111 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-metrics-client-ca\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.378506 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.376738 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.378506 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.377842 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-federate-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.378506 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.377842 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.378705 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.378567 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.378705 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.378686 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.380466 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:53.380441 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48eccb46_4917_49bb_b8f0_f40cf28de5f6.slice/crio-6a20ef282fdff3c3ed1d6f65c8c05a232f02e501e0c94acb02986b68934a92f2 WatchSource:0}: Error finding container 6a20ef282fdff3c3ed1d6f65c8c05a232f02e501e0c94acb02986b68934a92f2: Status 404 returned error can't find the container with id 6a20ef282fdff3c3ed1d6f65c8c05a232f02e501e0c94acb02986b68934a92f2 Nov 02 19:28:53.404917 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.404889 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:53.430144 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.430119 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsz7l\" (UniqueName: \"kubernetes.io/projected/e0180ace-54a9-4796-a2ab-f069f965573c-kube-api-access-zsz7l\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.443169 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.443146 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ssjs9"] Nov 02 19:28:53.446378 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:53.446353 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc967790c_1a44_4339_aa6e_8becad9f4474.slice/crio-c15b0ba630be0437aaa4115339eb08210c93f67fbe02732fa4edfd6156558d17 WatchSource:0}: Error finding container c15b0ba630be0437aaa4115339eb08210c93f67fbe02732fa4edfd6156558d17: Status 404 returned error can't find the container with id c15b0ba630be0437aaa4115339eb08210c93f67fbe02732fa4edfd6156558d17 Nov 02 19:28:53.532631 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.531678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:28:53.704075 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.704045 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-ckpzv"] Nov 02 19:28:53.707006 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:28:53.706978 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ba7edd_403e_423f_b9a6_92c08b7881a9.slice/crio-b0c475c8b9e175d9934ce73ce681b18a11ba3f321f88afa49ec6a1a196980a9a WatchSource:0}: Error finding container b0c475c8b9e175d9934ce73ce681b18a11ba3f321f88afa49ec6a1a196980a9a: Status 404 returned error can't find the container with id b0c475c8b9e175d9934ce73ce681b18a11ba3f321f88afa49ec6a1a196980a9a Nov 02 19:28:53.923243 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:53.923214 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57ddbd48bb-l5d62"] Nov 02 19:28:54.051940 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.051902 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" event={"ID":"e0180ace-54a9-4796-a2ab-f069f965573c","Type":"ContainerStarted","Data":"89590aeaf5245d153f1bbefae4e3f9adb43192d4db0960ddcd5a7d3ec0910213"} Nov 02 19:28:54.053424 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.053396 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ssjs9" event={"ID":"c967790c-1a44-4339-aa6e-8becad9f4474","Type":"ContainerStarted","Data":"c15b0ba630be0437aaa4115339eb08210c93f67fbe02732fa4edfd6156558d17"} Nov 02 19:28:54.054615 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.054592 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r8qkh" event={"ID":"13da2219-8d3c-4b28-b6b0-38b51415de22","Type":"ContainerStarted","Data":"c00e59c137866958842ee52e3b2741e2f170e25f718ad32b342a18ab108add41"} Nov 02 19:28:54.055767 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.055745 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tq4ks" event={"ID":"19ec87c5-1d05-4829-9a6d-cddcd0f84607","Type":"ContainerStarted","Data":"2f54c0ecffa913bcc377365538a402de44e285bd8b48c5aa16707496b8cb3a02"} Nov 02 19:28:54.057699 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.057682 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" event={"ID":"de1657c9-1cb9-44bb-b505-adf726ab2540","Type":"ContainerStarted","Data":"944d120ee8614ad526999c389a3ebce588c79153c8349b0f27ed5bd59521d85e"} Nov 02 19:28:54.057784 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.057735 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" event={"ID":"de1657c9-1cb9-44bb-b505-adf726ab2540","Type":"ContainerStarted","Data":"7776cb7bd1a5132a5fc536c216cb2ed321946410387f1c59bf5d42a0ecefd2dd"} Nov 02 19:28:54.057784 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.057762 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" event={"ID":"de1657c9-1cb9-44bb-b505-adf726ab2540","Type":"ContainerStarted","Data":"1fa3b1dbde50c225b153d860bda38b710d01a4612debb8bfb785683d0d1c5052"} Nov 02 19:28:54.058684 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.058665 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-ckpzv" event={"ID":"b5ba7edd-403e-423f-b9a6-92c08b7881a9","Type":"ContainerStarted","Data":"b0c475c8b9e175d9934ce73ce681b18a11ba3f321f88afa49ec6a1a196980a9a"} Nov 02 19:28:54.059846 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.059819 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" event={"ID":"48eccb46-4917-49bb-b8f0-f40cf28de5f6","Type":"ContainerStarted","Data":"6a20ef282fdff3c3ed1d6f65c8c05a232f02e501e0c94acb02986b68934a92f2"} Nov 02 19:28:54.149041 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.148971 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:54.343316 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:54.343234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:56.079589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:56.078419 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tq4ks" event={"ID":"19ec87c5-1d05-4829-9a6d-cddcd0f84607","Type":"ContainerStarted","Data":"0ad122aaea84cd9fe003cd547354bc467d41ebae920daabc33e946c7dbe93336"} Nov 02 19:28:56.079589 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:56.078555 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tq4ks" event={"ID":"19ec87c5-1d05-4829-9a6d-cddcd0f84607","Type":"ContainerStarted","Data":"1f677ebb415426a5cb9a88b24bf85dc241b46cb170526430e28b2bd55ecc0b9f"} Nov 02 19:28:56.085005 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:56.084966 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" event={"ID":"de1657c9-1cb9-44bb-b505-adf726ab2540","Type":"ContainerStarted","Data":"e23115f2bf893178a44c1b3190828869c901c26861572f8eb707226b5f9507e7"} Nov 02 19:28:56.085170 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:56.085050 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" event={"ID":"de1657c9-1cb9-44bb-b505-adf726ab2540","Type":"ContainerStarted","Data":"f4fb83bbacdbfc70e48f529a872efb96bbca5b011415e9676418c142ac4553f1"} Nov 02 19:28:56.085170 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:56.085092 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" event={"ID":"de1657c9-1cb9-44bb-b505-adf726ab2540","Type":"ContainerStarted","Data":"9e31a32939201ba6c7ad5f7a653e6aaa46f2eead25876e1682268f943c17cac1"} Nov 02 19:28:56.085976 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:56.085953 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:56.191852 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:56.191741 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tq4ks" podStartSLOduration=66.048305065 podStartE2EDuration="1m8.191722246s" totalImagesPullingTime="2.143417181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:56.188624995 +0000 UTC m=+69.027130871" watchObservedRunningTime="2026-11-02 19:28:56.191722246 +0000 UTC m=+69.030228122" Nov 02 19:28:56.332867 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:56.332804 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" podStartSLOduration=1.7621808429999999 podStartE2EDuration="6.33278247s" totalImagesPullingTime="4.570601627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:56.331750005 +0000 UTC m=+69.170255884" watchObservedRunningTime="2026-11-02 19:28:56.33278247 +0000 UTC m=+69.171288346" Nov 02 19:28:56.589236 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:56.588998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:57.093359 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:57.092757 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" event={"ID":"48eccb46-4917-49bb-b8f0-f40cf28de5f6","Type":"ContainerStarted","Data":"fa425181ecd461b8f5ab46ba9cd3331d76d2259dabcb6e753536437fee4faaa6"} Nov 02 19:28:57.097673 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:57.095638 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" event={"ID":"107e58b3-1281-416a-b713-8efc20576e9f","Type":"ContainerStarted","Data":"00022e7bfcbdf764f4301fccff5a0532097b37970be1af9d3dceb1bc1377b106"} Nov 02 19:28:57.097673 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:57.096687 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:57.103422 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:57.103368 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:28:57.108126 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:57.107822 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerStarted","Data":"1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a"} Nov 02 19:28:57.109046 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:57.108567 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerStarted","Data":"069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9"} Nov 02 19:28:57.203097 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:57.203044 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" podStartSLOduration=1.887945267 podStartE2EDuration="5.203028836s" totalImagesPullingTime="3.315083569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:57.202665206 +0000 UTC m=+70.041171084" watchObservedRunningTime="2026-11-02 19:28:57.203028836 +0000 UTC m=+70.041534714" Nov 02 19:28:57.315057 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:57.314287 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" podStartSLOduration=2.3181679170000002 podStartE2EDuration="5.314275069s" totalImagesPullingTime="2.996107152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:57.313222141 +0000 UTC m=+70.151728056" watchObservedRunningTime="2026-11-02 19:28:57.314275069 +0000 UTC m=+70.152780944" Nov 02 19:28:57.984808 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:57.984663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:58.121121 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.121092 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerStarted","Data":"80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1"} Nov 02 19:28:58.121556 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.121171 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerStarted","Data":"27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0"} Nov 02 19:28:58.121556 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.121198 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerStarted","Data":"d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d"} Nov 02 19:28:58.121556 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.121227 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerStarted","Data":"00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c"} Nov 02 19:28:58.127384 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.127141 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" event={"ID":"e0180ace-54a9-4796-a2ab-f069f965573c","Type":"ContainerStarted","Data":"cadd8f767873de3cf53d265a47b72d2e1705600b0225b26137d868cb0a95eb8f"} Nov 02 19:28:58.138857 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.138827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:58.142067 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.142044 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:28:58.240307 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.240247 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.894192664 podStartE2EDuration="9.240227643s" totalImagesPullingTime="5.331672404s" totalInitContainerRuntime="14.362575ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:58.237903848 +0000 UTC m=+71.076409751" watchObservedRunningTime="2026-11-02 19:28:58.240227643 +0000 UTC m=+71.078733518" Nov 02 19:28:58.744259 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.744159 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:58.907693 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:58.907605 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:28:59.134704 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:59.134665 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" event={"ID":"e0180ace-54a9-4796-a2ab-f069f965573c","Type":"ContainerStarted","Data":"4491ffa129f7ae3a8213964112c20c176ccdf5fac9d51bb39ce7c5d644732f2d"} Nov 02 19:28:59.135180 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:59.134736 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" event={"ID":"e0180ace-54a9-4796-a2ab-f069f965573c","Type":"ContainerStarted","Data":"5223fd5adc7a090f11ef55893bef5e4124523f831c567754ebce0c32ded0b28a"} Nov 02 19:28:59.248455 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:28:59.248394 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" podStartSLOduration=2.163083277 podStartE2EDuration="6.24838045s" totalImagesPullingTime="4.085297173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:59.248122608 +0000 UTC m=+72.086628484" watchObservedRunningTime="2026-11-02 19:28:59.24838045 +0000 UTC m=+72.086886330" Nov 02 19:29:00.142074 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.142038 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ssjs9" event={"ID":"c967790c-1a44-4339-aa6e-8becad9f4474","Type":"ContainerStarted","Data":"b06d594f5d36f0037a5b6630e4ceadba5160af01f8dd1a3dce69bd300126c1d1"} Nov 02 19:29:00.142516 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.142152 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:29:00.252908 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.252857 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ssjs9" podStartSLOduration=65.965169038 podStartE2EDuration="1m12.25284203s" totalImagesPullingTime="6.287672992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:29:00.250955013 +0000 UTC m=+73.089460887" watchObservedRunningTime="2026-11-02 19:29:00.25284203 +0000 UTC m=+73.091347904" Nov 02 19:29:00.597553 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.597149 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cb5d7db54-whf7r"] Nov 02 19:29:00.604531 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.604497 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.669735 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.669706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zj4k\" (UniqueName: \"kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.669909 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.669742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.669909 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.669765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.669909 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.669812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.669909 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.669832 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.669909 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.669848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.669909 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.669877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.703335 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.703299 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 19:29:00.703550 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.703400 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 19:29:00.703550 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.703443 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 19:29:00.703550 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.703453 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2gz6n\"" Nov 02 19:29:00.703550 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.703529 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 19:29:00.703550 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.703552 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 19:29:00.705642 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.705614 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cb5d7db54-whf7r"] Nov 02 19:29:00.770520 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.770444 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.770698 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.770555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zj4k\" (UniqueName: \"kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.770698 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.770581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.770698 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.770611 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.770698 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.770679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.770911 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.770702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.770911 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.770720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.772088 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.771972 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.772088 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.771981 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.772466 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.772425 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.772775 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.772726 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.773872 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.773851 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.774757 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.774721 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.826903 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.826865 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zj4k\" (UniqueName: \"kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.923591 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.923553 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:00.948622 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:00.948521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:01.119017 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:01.118982 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:01.303337 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:01.303305 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cb5d7db54-whf7r"] Nov 02 19:29:01.337676 ip-10-0-131-244 kubenswrapper[2573]: W1102 19:29:01.337649 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2752b72d_0a38_4fdd_ab0f_905bae22cedf.slice/crio-6825ab200ababe5c3a84c253175e259e494c1f731647bf9841faef4a8f98c790 WatchSource:0}: Error finding container 6825ab200ababe5c3a84c253175e259e494c1f731647bf9841faef4a8f98c790: Status 404 returned error can't find the container with id 6825ab200ababe5c3a84c253175e259e494c1f731647bf9841faef4a8f98c790 Nov 02 19:29:01.584913 ip-10-0-131-244 kubenswrapper[2573]: I1102 19:29:01.584560 2573 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:29:01.584772 ip-10-0-131-244 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 19:29:01.596049 ip-10-0-131-244 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 19:29:01.596341 ip-10-0-131-244 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 19:29:01.596442 ip-10-0-131-244 systemd[1]: kubelet.service: Consumed 4.721s CPU time. Nov 02 19:29:01.609006 ip-10-0-131-244 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:29:01.689639 ip-10-0-131-244 kubenswrapper[5850]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:29:01.689639 ip-10-0-131-244 kubenswrapper[5850]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:29:01.689639 ip-10-0-131-244 kubenswrapper[5850]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:29:01.689639 ip-10-0-131-244 kubenswrapper[5850]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:29:01.691778 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691685 5850 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:29:01.691778 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691778 5850 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:29:01.691778 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691782 5850 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691787 5850 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691790 5850 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691793 5850 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691796 5850 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691800 5850 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691803 5850 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691805 5850 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691808 5850 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691811 5850 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691814 5850 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691817 5850 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691819 5850 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691822 5850 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691824 5850 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691827 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691830 5850 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691832 5850 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691835 5850 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691838 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:29:01.691876 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691841 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691843 5850 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691846 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691849 5850 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691864 5850 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691867 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691870 5850 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691873 5850 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691875 5850 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691879 5850 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691882 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691885 5850 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691887 5850 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691890 5850 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691893 5850 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691896 5850 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691898 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691901 5850 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691903 5850 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:29:01.692347 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691906 5850 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691908 5850 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691911 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691913 5850 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691916 5850 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691918 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691921 5850 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691923 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691926 5850 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691928 5850 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691931 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691933 5850 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691936 5850 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691938 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691942 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691945 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691947 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691950 5850 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691953 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691957 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:29:01.692808 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691959 5850 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691962 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691964 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691967 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691969 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691972 5850 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691975 5850 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691978 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691981 5850 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691983 5850 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691986 5850 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691988 5850 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691991 5850 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691994 5850 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691996 5850 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.691999 5850 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692001 5850 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692004 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692008 5850 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:29:01.693272 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692012 5850 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692015 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692018 5850 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692021 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692024 5850 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692026 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692029 5850 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692032 5850 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692035 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692038 5850 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692041 5850 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692044 5850 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692049 5850 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692052 5850 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692055 5850 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692058 5850 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692062 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692064 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692067 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:29:01.693741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692070 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692072 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692075 5850 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692077 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692080 5850 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692083 5850 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692085 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692925 5850 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692934 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692937 5850 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692941 5850 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692944 5850 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692947 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692950 5850 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692953 5850 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692955 5850 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692958 5850 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692961 5850 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692963 5850 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692966 5850 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:29:01.694186 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692970 5850 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692973 5850 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692975 5850 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692978 5850 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692981 5850 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692983 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692987 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692990 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692993 5850 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692995 5850 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.692998 5850 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693001 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693005 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693009 5850 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693013 5850 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693017 5850 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693021 5850 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693025 5850 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693030 5850 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693034 5850 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:29:01.694703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693038 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693042 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693047 5850 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693050 5850 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693052 5850 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693055 5850 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693058 5850 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693060 5850 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693063 5850 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693066 5850 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693068 5850 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693071 5850 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693074 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693077 5850 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693080 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693082 5850 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693085 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693088 5850 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693091 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693094 5850 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:29:01.695170 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693097 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693100 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693102 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693105 5850 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693107 5850 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693110 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693112 5850 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693115 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693118 5850 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693120 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693123 5850 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693125 5850 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693128 5850 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693131 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693133 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693136 5850 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693138 5850 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693143 5850 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:29:01.695719 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693148 5850 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693152 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693154 5850 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693157 5850 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693161 5850 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693165 5850 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693168 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693171 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693174 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693177 5850 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693179 5850 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693182 5850 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693185 5850 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693187 5850 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693191 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693194 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693197 5850 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693200 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693202 5850 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693205 5850 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:29:01.696142 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693207 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693210 5850 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693212 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693215 5850 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693218 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693220 5850 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693223 5850 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693226 5850 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693228 5850 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693231 5850 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693233 5850 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693238 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693240 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693243 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693245 5850 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693868 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693876 5850 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693880 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693883 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693886 5850 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:29:01.696642 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693889 5850 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693892 5850 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693895 5850 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693898 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693901 5850 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693903 5850 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693907 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693910 5850 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693912 5850 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693916 5850 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693920 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693923 5850 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693926 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693929 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693931 5850 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693934 5850 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693936 5850 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693939 5850 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693942 5850 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:29:01.697105 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693944 5850 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693950 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693953 5850 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693956 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693959 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693962 5850 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693964 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693967 5850 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693970 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693972 5850 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693975 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693978 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693981 5850 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693984 5850 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693986 5850 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693989 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693991 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693994 5850 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693997 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:29:01.697569 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.693999 5850 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694002 5850 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694005 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694008 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694011 5850 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694013 5850 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694016 5850 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694019 5850 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694021 5850 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694024 5850 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694027 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694030 5850 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694033 5850 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694035 5850 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694040 5850 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694042 5850 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694045 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694048 5850 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694050 5850 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694053 5850 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:29:01.698036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694056 5850 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694059 5850 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694061 5850 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694064 5850 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694066 5850 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694069 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694073 5850 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694076 5850 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694080 5850 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694082 5850 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694085 5850 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694089 5850 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694091 5850 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694094 5850 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694097 5850 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694100 5850 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694103 5850 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694106 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694109 5850 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:29:01.698534 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694111 5850 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694114 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694116 5850 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694119 5850 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694122 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694124 5850 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694127 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694131 5850 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694133 5850 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694136 5850 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694138 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694141 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694143 5850 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694146 5850 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694149 5850 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694151 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694154 5850 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694156 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694159 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694161 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:29:01.698984 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694164 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694166 5850 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694169 5850 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.694172 5850 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694266 5850 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694274 5850 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694280 5850 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694285 5850 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694290 5850 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694294 5850 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694298 5850 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694303 5850 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694306 5850 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694309 5850 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694313 5850 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694316 5850 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694320 5850 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694323 5850 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694326 5850 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694337 5850 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694341 5850 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694344 5850 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694354 5850 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694357 5850 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:29:01.699456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694360 5850 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694363 5850 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694367 5850 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694371 5850 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694374 5850 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694378 5850 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694382 5850 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694385 5850 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694388 5850 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694391 5850 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694394 5850 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694397 5850 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694401 5850 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694404 5850 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694408 5850 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694411 5850 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694414 5850 flags.go:64] FLAG: --enable-server="true" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694417 5850 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694425 5850 flags.go:64] FLAG: --event-burst="100" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694428 5850 flags.go:64] FLAG: --event-qps="50" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694431 5850 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694434 5850 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694437 5850 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694442 5850 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:29:01.700042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694445 5850 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694448 5850 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694451 5850 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694456 5850 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694459 5850 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694462 5850 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694465 5850 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694468 5850 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694492 5850 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694497 5850 flags.go:64] FLAG: --feature-gates="" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694503 5850 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694508 5850 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694513 5850 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694516 5850 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694520 5850 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694523 5850 flags.go:64] FLAG: --help="false" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694526 5850 flags.go:64] FLAG: --hostname-override="ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694529 5850 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694533 5850 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694536 5850 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694539 5850 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694542 5850 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694546 5850 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:29:01.701125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694548 5850 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694551 5850 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694554 5850 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694558 5850 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694561 5850 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694564 5850 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694568 5850 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694571 5850 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694574 5850 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694577 5850 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694579 5850 flags.go:64] FLAG: --lock-file="" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694583 5850 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694587 5850 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694590 5850 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694596 5850 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694599 5850 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694602 5850 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694605 5850 flags.go:64] FLAG: --logging-format="text" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694608 5850 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694612 5850 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694615 5850 flags.go:64] FLAG: --manifest-url="" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694618 5850 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694622 5850 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694627 5850 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694631 5850 flags.go:64] FLAG: --max-pods="110" Nov 02 19:29:01.702058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694634 5850 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694637 5850 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694641 5850 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694644 5850 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694647 5850 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694650 5850 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694652 5850 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694660 5850 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694663 5850 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694666 5850 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694670 5850 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694673 5850 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694676 5850 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694679 5850 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694682 5850 flags.go:64] FLAG: --port="10250" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694685 5850 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694688 5850 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e490ebd6853ea3b9" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694691 5850 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694694 5850 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694699 5850 flags.go:64] FLAG: --register-node="true" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694702 5850 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694706 5850 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694709 5850 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694712 5850 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694715 5850 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694719 5850 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:29:01.703188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694722 5850 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694725 5850 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694729 5850 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694732 5850 flags.go:64] FLAG: --runonce="false" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694736 5850 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694739 5850 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694742 5850 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694745 5850 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694748 5850 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694752 5850 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694755 5850 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694758 5850 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694761 5850 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694764 5850 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694766 5850 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694769 5850 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694772 5850 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694775 5850 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694779 5850 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694784 5850 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694787 5850 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694790 5850 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694797 5850 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694808 5850 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694811 5850 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:29:01.704259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694816 5850 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694819 5850 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694822 5850 flags.go:64] FLAG: --v="2" Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694827 5850 flags.go:64] FLAG: --version="false" Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694831 5850 flags.go:64] FLAG: --vmodule="" Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694836 5850 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.694839 5850 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695038 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695043 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695047 5850 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695050 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695055 5850 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695058 5850 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695060 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695063 5850 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695066 5850 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695068 5850 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695071 5850 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695074 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695077 5850 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695079 5850 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:29:01.705321 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695082 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695084 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695087 5850 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695089 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695092 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695100 5850 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695103 5850 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695106 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695108 5850 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695111 5850 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695113 5850 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695117 5850 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695120 5850 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695123 5850 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695126 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695128 5850 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695131 5850 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695134 5850 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695136 5850 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:29:01.706366 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695145 5850 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695147 5850 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695151 5850 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695156 5850 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695159 5850 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695162 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695164 5850 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695167 5850 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695169 5850 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695172 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695175 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695178 5850 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695180 5850 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695183 5850 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695186 5850 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695189 5850 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695192 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695195 5850 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695197 5850 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:29:01.707177 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695206 5850 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695209 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695211 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695214 5850 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695217 5850 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695221 5850 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695223 5850 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695226 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695229 5850 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695231 5850 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695234 5850 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695237 5850 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695240 5850 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695242 5850 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695245 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695247 5850 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695251 5850 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695255 5850 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695259 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:29:01.707994 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695262 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695265 5850 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695268 5850 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695270 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695273 5850 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695276 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695278 5850 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695281 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695284 5850 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695286 5850 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695289 5850 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695292 5850 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695294 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695296 5850 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695306 5850 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695310 5850 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695313 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695315 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695322 5850 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:29:01.708682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695324 5850 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695327 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695330 5850 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695333 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695335 5850 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695338 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695340 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695343 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695346 5850 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695348 5850 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695352 5850 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695355 5850 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695357 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695360 5850 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695363 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695365 5850 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:29:01.709317 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.695393 5850 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695434 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695439 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695442 5850 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695444 5850 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695447 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695450 5850 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695452 5850 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695455 5850 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695458 5850 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695460 5850 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695463 5850 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695490 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695495 5850 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695500 5850 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695503 5850 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695508 5850 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695510 5850 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695513 5850 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:29:01.710009 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695517 5850 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695520 5850 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695524 5850 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695526 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695529 5850 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695532 5850 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695535 5850 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695540 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695543 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695546 5850 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695549 5850 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695552 5850 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695555 5850 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695558 5850 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695561 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695564 5850 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695568 5850 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695570 5850 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695573 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695576 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:29:01.710874 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695578 5850 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695581 5850 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695583 5850 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695586 5850 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695589 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695591 5850 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695600 5850 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695603 5850 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695606 5850 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695610 5850 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695613 5850 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695615 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695618 5850 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695621 5850 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695625 5850 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695630 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695633 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695636 5850 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695638 5850 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:29:01.711738 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695642 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695646 5850 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695648 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695651 5850 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695653 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695656 5850 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695659 5850 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695661 5850 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695664 5850 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695666 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695669 5850 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695671 5850 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695674 5850 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695677 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695679 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695682 5850 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695685 5850 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695687 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695690 5850 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:29:01.712285 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695692 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695701 5850 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695704 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695708 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695711 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695713 5850 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695716 5850 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695719 5850 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695735 5850 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695738 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695742 5850 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695745 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695748 5850 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695752 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695755 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695757 5850 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695760 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695762 5850 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695765 5850 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695768 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:29:01.712824 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695770 5850 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695773 5850 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695776 5850 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695779 5850 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695782 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695785 5850 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695787 5850 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695790 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695793 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.695795 5850 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.700769 5850 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.700820 5850 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.700846 5850 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.700880 5850 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701350 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701369 5850 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701376 5850 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701381 5850 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701386 5850 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701390 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:29:01.713432 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701395 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701399 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701404 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701408 5850 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701412 5850 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701417 5850 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701421 5850 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701426 5850 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701430 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701434 5850 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701440 5850 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701444 5850 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701449 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701455 5850 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701460 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701465 5850 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701484 5850 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701490 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701494 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701499 5850 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:29:01.713974 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701504 5850 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701508 5850 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701513 5850 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701517 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701522 5850 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701526 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701530 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701537 5850 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701541 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701546 5850 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701550 5850 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701554 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701559 5850 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701564 5850 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701569 5850 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701573 5850 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701577 5850 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701580 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701584 5850 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701589 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:29:01.714465 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701593 5850 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701597 5850 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701602 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701606 5850 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701610 5850 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701618 5850 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701624 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701629 5850 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701634 5850 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701639 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701643 5850 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701647 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701652 5850 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701656 5850 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701660 5850 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701665 5850 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701669 5850 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701673 5850 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701677 5850 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:29:01.715006 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701682 5850 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701688 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701693 5850 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701697 5850 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701702 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701707 5850 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701712 5850 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701716 5850 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701720 5850 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701725 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701729 5850 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701734 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701739 5850 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701743 5850 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701748 5850 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701752 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701756 5850 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701762 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701766 5850 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701771 5850 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:29:01.715506 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701775 5850 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701779 5850 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701784 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701789 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701793 5850 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701797 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701804 5850 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701809 5850 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701814 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701819 5850 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701823 5850 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701828 5850 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701833 5850 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701838 5850 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701843 5850 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701847 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701852 5850 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701857 5850 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701861 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701865 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:29:01.716087 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.701870 5850 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.701915 5850 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702196 5850 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702205 5850 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702210 5850 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702214 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702219 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702232 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702238 5850 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702242 5850 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702246 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702251 5850 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702255 5850 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702259 5850 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702264 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:29:01.716621 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702268 5850 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702272 5850 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702277 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702281 5850 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702285 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702289 5850 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702294 5850 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702299 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702303 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702308 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702312 5850 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702317 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702322 5850 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702326 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702331 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702335 5850 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702339 5850 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702344 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702348 5850 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:29:01.716980 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702353 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702357 5850 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702361 5850 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702366 5850 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702371 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702375 5850 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702381 5850 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702386 5850 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702391 5850 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702395 5850 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702399 5850 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702403 5850 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702408 5850 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702412 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702416 5850 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702421 5850 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702425 5850 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702429 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702434 5850 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:29:01.717434 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702438 5850 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702443 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702447 5850 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702454 5850 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702462 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702467 5850 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702493 5850 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702497 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702502 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702506 5850 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702510 5850 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702515 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702520 5850 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702524 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702529 5850 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702533 5850 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702537 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702542 5850 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702546 5850 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:29:01.717943 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702550 5850 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702555 5850 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702560 5850 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702564 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702568 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702572 5850 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702576 5850 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702581 5850 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702585 5850 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702590 5850 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702594 5850 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702599 5850 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702603 5850 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702610 5850 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702615 5850 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702621 5850 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702626 5850 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702631 5850 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702635 5850 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702639 5850 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:29:01.718401 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702644 5850 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702648 5850 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702653 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702657 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702662 5850 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702666 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702670 5850 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702675 5850 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702679 5850 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702684 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702688 5850 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702693 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702697 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702701 5850 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702707 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702711 5850 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:29:01.718921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.702758 5850 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702820 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702828 5850 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702833 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702838 5850 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702842 5850 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702847 5850 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702851 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702856 5850 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702861 5850 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702865 5850 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702869 5850 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702874 5850 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702879 5850 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702884 5850 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702888 5850 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702893 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702897 5850 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702902 5850 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:29:01.719348 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702907 5850 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702911 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702915 5850 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702919 5850 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702924 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702928 5850 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702935 5850 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702943 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702949 5850 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702953 5850 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702958 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702962 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702967 5850 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702971 5850 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702976 5850 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702981 5850 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702986 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702990 5850 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702994 5850 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:29:01.719828 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.702998 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703003 5850 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703007 5850 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703011 5850 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703016 5850 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703020 5850 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703025 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703029 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703034 5850 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703039 5850 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703043 5850 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703047 5850 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703051 5850 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703056 5850 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703060 5850 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703064 5850 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703072 5850 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703078 5850 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703082 5850 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703087 5850 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:29:01.720284 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703091 5850 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703095 5850 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703100 5850 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703104 5850 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703109 5850 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703113 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703119 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703124 5850 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703128 5850 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703132 5850 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703136 5850 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703141 5850 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703148 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703152 5850 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703157 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703161 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703165 5850 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703170 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703174 5850 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703179 5850 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:29:01.720836 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703183 5850 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703187 5850 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703191 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703196 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703200 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703205 5850 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703208 5850 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703212 5850 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703216 5850 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703221 5850 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703225 5850 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703229 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703234 5850 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703238 5850 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703242 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703247 5850 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703251 5850 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703256 5850 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703260 5850 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:29:01.721368 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703265 5850 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703270 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703274 5850 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703279 5850 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703283 5850 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703288 5850 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703294 5850 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703298 5850 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703302 5850 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:01.703306 5850 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.703402 5850 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.705014 5850 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.705091 5850 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.705357 5850 server.go:1014] "Starting client certificate rotation" Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.705509 5850 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.705583 5850 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:22:48 +0000 UTC" deadline="2028-06-27 22:05:17.01406089 +0000 UTC" Nov 02 19:29:01.721879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.705612 5850 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14474h36m15.308450397s" Nov 02 19:29:01.722460 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.706111 5850 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:29:01.722460 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.708036 5850 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:29:01.722460 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.709310 5850 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:29:01.722460 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.711862 5850 log.go:25] "Validated CRI v1 image API" Nov 02 19:29:01.722460 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.712617 5850 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:29:01.722460 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.715450 5850 fs.go:135] Filesystem UUIDs: map[3ab770dc-49a2-4bf6-939d-61d9ac9e989e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7b653955-db88-4b7d-8c76-f16bd4af0e76:/dev/nvme0n1p4] Nov 02 19:29:01.722776 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.715492 5850 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/010fac1e5b127921d19fae7d389c4501ef76dcada819b5030e50cfed167af910/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/010fac1e5b127921d19fae7d389c4501ef76dcada819b5030e50cfed167af910/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f5bd0153977ea80cbdeebb54fbaafe9846cf48385a69ea2c02d3dd89d04e4ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f5bd0153977ea80cbdeebb54fbaafe9846cf48385a69ea2c02d3dd89d04e4ec/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/125e3cca30e0c0415432b25dca84e50a35329edce80ff1b0b3d0ddc3690bf18b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/125e3cca30e0c0415432b25dca84e50a35329edce80ff1b0b3d0ddc3690bf18b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2573d463444e98c321960ffca385dbadae267636744405fc811321d91d50db65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2573d463444e98c321960ffca385dbadae267636744405fc811321d91d50db65/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d9a6ad973d3cc9f414d9443c94cbd1984d66af12b8401a86ade6ca6dd2388b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d9a6ad973d3cc9f414d9443c94cbd1984d66af12b8401a86ade6ca6dd2388b8/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f54c0ecffa913bcc377365538a402de44e285bd8b48c5aa16707496b8cb3a02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f54c0ecffa913bcc377365538a402de44e285bd8b48c5aa16707496b8cb3a02/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/436a8b4fe9ec3b4600bfa808f8f091d66e6f712709acf0841b833d5c3d55fc79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/436a8b4fe9ec3b4600bfa808f8f091d66e6f712709acf0841b833d5c3d55fc79/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47a581c4ec21a6c6ed42132dfa90f4296f0e49de45b8537352e115e71a010a86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47a581c4ec21a6c6ed42132dfa90f4296f0e49de45b8537352e115e71a010a86/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/488c6e0df5e686add29118b2f232ca1d1e0f5e218f2361ce793f0b0934bde8b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/488c6e0df5e686add29118b2f232ca1d1e0f5e218f2361ce793f0b0934bde8b2/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48fe43dd697504c22f3b846239e421a443aac2854ece9de26c4c8661810f20e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48fe43dd697504c22f3b846239e421a443aac2854ece9de26c4c8661810f20e6/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6446e2aebac2a76a71da81fe3d07269e7e2b1f4deef71b0608e84f70b5c6f130/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6446e2aebac2a76a71da81fe3d07269e7e2b1f4deef71b0608e84f70b5c6f130/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6825ab200ababe5c3a84c253175e259e494c1f731647bf9841faef4a8f98c790/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6825ab200ababe5c3a84c253175e259e494c1f731647bf9841faef4a8f98c790/userdata/shm major:0 minor:497 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a20ef282fdff3c3ed1d6f65c8c05a232f02e501e0c94acb02986b68934a92f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a20ef282fdff3c3ed1d6f65c8c05a232f02e501e0c94acb02986b68934a92f2/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/770569fc3162f6a8fa8c1dc5e649325d3108c12182d41839520c33c02428e386/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/770569fc3162f6a8fa8c1dc5e649325d3108c12182d41839520c33c02428e386/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/884442753f0436d3d0b3c66bf30628040932f27f63cc660173957f4be66631ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/884442753f0436d3d0b3c66bf30628040932f27f63cc660173957f4be66631ae/userdata/shm major:0 minor:345 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/888426523d429d534a23b252e334071e1313754d18713d8f3dbaf7658ddee31c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/888426523d429d534a23b252e334071e1313754d18713d8f3dbaf7658ddee31c/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88f4fa50454796aaf822bc3612d51da1c57f8ba5155b13a1e488fd4da2a95df1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88f4fa50454796aaf822bc3612d51da1c57f8ba5155b13a1e488fd4da2a95df1/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89590aeaf5245d153f1bbefae4e3f9adb43192d4db0960ddcd5a7d3ec0910213/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89590aeaf5245d153f1bbefae4e3f9adb43192d4db0960ddcd5a7d3ec0910213/userdata/shm major:0 minor:394 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dfcdfe17db9bc3ac0a2af09c527fb7c32bdf8b17eb49aad1719f1e8638a3a8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dfcdfe17db9bc3ac0a2af09c527fb7c32bdf8b17eb49aad1719f1e8638a3a8a/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d733afefb0412f2afa49be26fbc7502b827d31df450cccad406a2c401b73a9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d733afefb0412f2afa49be26fbc7502b827d31df450cccad406a2c401b73a9a/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0c475c8b9e175d9934ce73ce681b18a11ba3f321f88afa49ec6a1a196980a9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0c475c8b9e175d9934ce73ce681b18a11ba3f321f88afa49ec6a1a196980a9a/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd10fc894d0a3082e0098d4444e6cd275aa026871f299331bcc904c7ccd6ea53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd10fc894d0a3082e0098d4444e6cd275aa026871f299331bcc904c7ccd6ea53/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bfc75af335b31db6f21ecab7e09792756a3e511287d837b8275c12a1c04788d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bfc75af335b31db6f21ecab7e09792756a3e511287d837b8275c12a1c04788d3/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c00e59c137866958842ee52e3b2741e2f170e25f718ad32b342a18ab108add41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c00e59c137866958842ee52e3b2741e2f170e25f718ad32b342a18ab108add41/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c15b0ba630be0437aaa4115339eb08210c93f67fbe02732fa4edfd6156558d17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c15b0ba630be0437aaa4115339eb08210c93f67fbe02732fa4edfd6156558d17/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd2a4d606f0a0c4df49062b4adcf1ff50aa7c43406c4a7c6b0ff5e023a05051a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd2a4d606f0a0c4df49062b4adcf1ff50aa7c43406c4a7c6b0ff5e023a05051a/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb59f3c761721d25c75ceac6d48db30c50ac93b0fe5361d3b199e33e313c4799/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb59f3c761721d25c75ceac6d48db30c50ac93b0fe5361d3b199e33e313c4799/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/107e58b3-1281-416a-b713-8efc20576e9f/volumes/kubernetes.io~projected/kube-api-access-dvb8m:{mountpoint:/var/lib/kubelet/pods/107e58b3-1281-416a-b713-8efc20576e9f/volumes/kubernetes.io~projected/kube-api-access-dvb8m major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/107e58b3-1281-416a-b713-8efc20576e9f/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/107e58b3-1281-416a-b713-8efc20576e9f/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13da2219-8d3c-4b28-b6b0-38b51415de22/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/13da2219-8d3c-4b28-b6b0-38b51415de22/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19ec87c5-1d05-4829-9a6d-cddcd0f84607/volumes/kubernetes.io~projected/kube-api-access-n9lvp:{mountpoint:/var/lib/kubelet/pods/19ec87c5-1d05-4829-9a6d-cddcd0f84607/volumes/kubernetes.io~projected/kube-api-access-n9lvp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19ec87c5-1d05-4829-9a6d-cddcd0f84607/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/19ec87c5-1d05-4829-9a6d-cddcd0f84607/volumes/kubernetes.io~secret/metrics-certs major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24100d48-fca9-4735-81a6-a2455bf00804/volumes/kubernetes.io~projected/kube-api-access-5z4dh:{mountpoint:/var/lib/kubelet/pods/24100d48-fca9-4735-81a6-a2455bf00804/volumes/kubernetes.io~projected/kube-api-access-5z4dh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2602c042-a534-4767-8bb3-e5d2b93b3a62/volumes/kubernetes.io~projected/kube-api-access-vsg8n:{mountpoint:/var/lib/kubelet/pods/2602c042-a534-4767-8bb3-e5d2b93b3a62/volumes/kubernetes.io~projected/kube-api-access-vsg8n major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes/kubernetes.io~projected/kube-api-access-5zj4k:{mountpoint:/var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes/kubernetes.io~projected/kube-api-access-5zj4k major:0 minor:496 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:495 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:490 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2867a688-7f22-4419-a69c-aef174888edc/volumes/kubernetes.io~projected/kube-api-access-dl8kl:{mountpoint:/var/lib/kubelet/pods/2867a688-7f22-4419-a69c-aef174888edc/volumes/kubernetes.io~projected/kube-api-access-dl8kl major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2867a688-7f22-4419-a69c-aef174888edc/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2867a688-7f22-4419-a69c-aef174888edc/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2867a688-7f22-4419-a69c-aef174888edc/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/2867a688-7f22-4419-a69c-aef174888edc/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/405d3ed6-8332-46f0-b97d-326543807323/volumes/kubernetes.io~projected/kube-api-access-q6j95:{mountpoint:/var/lib/kubelet/pods/405d3ed6-8332-46f0-b97d-326543807323/volumes/kubernetes.io~projected/kube-api-access-q6j95 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/405d3ed6-8332-46f0-b97d-326543807323/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/405d3ed6-8332-46f0-b97d-326543807323/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42a08de9-3178-4d75-87b7-0cea7a813dd3/volumes/kubernetes.io~projected/kube-api-access-tq5p9:{mountpoint:/var/lib/kubelet/pods/42a08de9-3178-4d75-87b7-0cea7a813dd3/volumes/kubernetes.io~projected/kube-api-access-tq5p9 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43a8f63a-57a0-4be5-b55f-621b67342950/volumes/kubernetes.io~projected/kube-api-access-bgsrp:{mountpoint:/var/lib/kubelet/pods/43a8f63a-57a0-4be5-b55f-621b67342950/volumes/kubernetes.io~projected/kube-api-access-bgsrp major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43a8f63a-57a0-4be5-b55f-621b67342950/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/43a8f63a-57a0-4be5-b55f-621b67342950/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43a8f63a-57a0-4be5-b55f-621b67342950/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/43a8f63a-57a0-4be5-b55f-621b67342950/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~projected/kube-api-access-kjvdn:{mountpoint:/var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~projected/kube-api-access-kjvdn major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53fc9047-2f91-4806-acfd-d23b1b0cbc71/volumes/kubernetes.io~projected/kube-api-access-s6f7f:{mountpoint:/var/lib/kubelet/pods/53fc9047-2f91-4806-acfd-d23b1b0cbc71/volumes/kubernetes.io~projected/kube-api-access-s6f7f major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7acda1fa-9e95-4357-ba7b-7315edb4352e/volumes/kubernetes.io~projected/kube-api-access-qcgb7:{mountpoint:/var/lib/kubelet/pods/7acda1fa-9e95-4357-ba7b-7315edb4352e/volumes/kubernetes.io~projected/kube-api-access-qcgb7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7acda1fa-9e95-4357-ba7b-7315edb4352e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7acda1fa-9e95-4357-ba7b-7315edb4352e/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f/volumes/kubernetes.io~projected/kube-api-access-28kv4:{mountpoint:/var/lib/kubelet/pods/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f/volumes/kubernetes.io~projected/kube-api-access-28kv4 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d9559a8-3f33-4ea2-b478-8b9474d99ed7/volumes/kubernetes.io~projected/kube-api-access-d5bhk:{mountpoint:/var/lib/kubelet/pods/9d9559a8-3f33-4ea2-b478-8b9474d99ed7/volumes/kubernetes.io~projected/kube-api-access-d5bhk major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2/volumes/kubernetes.io~projected/kube-api-access-54czs:{mountpoint:/var/lib/kubelet/pods/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2/volumes/kubernetes.io~projected/kube-api-access-54czs major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2de0d2d-3838-4979-b2ec-c3ed7df1beb9/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/a2de0d2d-3838-4979-b2ec-c3ed7df1beb9/volumes/kubernetes.io~secret/tls-certificates major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/web-config/alertmanager/9 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/web-config/config-reloader/6 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~empty-dir/config-out major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~projected/kube-api-access-bpzm9:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~projected/kube-api-access-bpzm9 major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~projected/tls-assets major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/config-volume major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/web-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ba7edd-403e-423f-b9a6-92c08b7881a9/volumes/kubernetes.io~projected/kube-api-access-7rkdj:{mountpoint:/var/lib/kubelet/pods/b5ba7edd-403e-423f-b9a6-92c08b7881a9/volumes/kubernetes.io~projected/kube-api-access-7rkdj major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c967790c-1a44-4339-aa6e-8becad9f4474/volumes/kubernetes.io~projected/kube-api-access-fjqwf:{mountpoint:/var/lib/kubelet/pods/c967790c-1a44-4339-aa6e-8becad9f4474/volumes/kubernetes.io~projected/kube-api-access-fjqwf major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d71e26dd-fe22-44a6-9b04-bf30e4aef71a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d71e26dd-fe22-44a6-9b04-bf30e4aef71a/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daf0d624-3b0c-4ceb-897a-c9d8b54213e2/volumes/kubernetes.io~projected/kube-api-access-r6828:{mountpoint:/var/lib/kubelet/pods/daf0d624-3b0c-4ceb-897a-c9d8b54213e2/volumes/kubernetes.io~projected/kube-api-access-r6828 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daf0d624-3b0c-4ceb-897a-c9d8b54213e2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/daf0d624-3b0c-4ceb-897a-c9d8b54213e2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daf0d624-3b0c-4ceb-897a-c9d8b54213e2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/daf0d624-3b0c-4ceb-897a-c9d8b54213e2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97/volumes/kubernetes.io~projected/kube-api-access-hhdz2:{mountpoint:/var/lib/kubelet/pods/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97/volumes/kubernetes.io~projected/kube-api-access-hhdz2 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~projected/kube-api-access-5nhrj:{mountpoint:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~projected/kube-api-access-5nhrj major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~projected/kube-api-access-zsz7l:{mountpoint:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~projected/kube-api-access-zsz7l major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f31b231b-c403-428b-a1a3-f516644ac9e5/volumes/kubernetes.io~projected/kube-api-access-sb6f4:{mountpoint:/var/lib/kubelet/pods/f31b231b-c403-428b-a1a3-f516644ac9e5/volumes/kubernetes.io~projected/kube-api-access-sb6f4 major:0 minor:51 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3b17a181f891ea0f0361e8b2fb6c171862eb0172325e6d40fa7b486e16432cd8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6a2693bd546fbb3814f355b63b3f6946432df53261f96e2c3754a1ed95fd912e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c403cea473a1a3b89d6da5263e4f98ae962413f7cf8c641e8d2a8f91adb33fab/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a788a156162bb4619597df68a2a0c11e5059445d8d9859a5027ae5cf32c8be07/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dae2ee62586607e5f38b1466b4351e9e0993b86690cc6d8d21f6d7a5cabefa8d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bc4ba54ce01e6cc87652e2ea9e5e00e551f91f12826a25d5800c6f6a4dfd8652/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/5f864b9bf95c0016e4dc9e0ebec6bc25371abaf77e3ad68121e8f501b9964fb7/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b3714cd036ec3e257d04701c1a99f2299e151144487f315dc8abd2e507d6150b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d6611c430501c6d323d601c72a51f594505aa837a30fe18d338ef95b11ec5f3c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/467b834e6e4799c0196e9731b9521115b237734847754f23f099fe250691d36b/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/d25c3f0ac13d8053e91783f67e62e46e3a17bd2b016870b03366e30c9c54d161/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/a9f24a70961dbd3d99a6621ac41839d4cd0443781c04e4be1c5ebd2c6e4f0656/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/83edf87476487ea9d814f3827d5e8b50e5638e128e2bc25e505597505be0c625/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/20ccd52a277dd4c0c9821d5fd9937700094985ca94ea7ffd450b797395156201/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/286d9b5cae9052ee76104e68da2284009cd4605a55f0dde3e2dfd1864b1635d5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9ae08a44046d98b31bc925a782e10ebe25be97d4b9b470d16ab1bf027fcbca73/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5a597450302c361bb16d181fa95c23d3208f236c936de9185f93d68851398c22/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4be046a9cc6746897be1d1ff9f4ac3367a215b870f2a35d34ea165c9c687e518/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8b153d43a6aa63a3cb2fba0003c7b7e1aecd754082688e1252ce98ce549ace4f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f8341da8061e3b6243021a90c32d425f4df116af50c504130fe80690101d00cb/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/635ca86e8308d57dc83f6e183f66f5bc456e49a504d75e66bf3a5c5b3bf6d475/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/a6686e2c5cd32d3e8f1c1341c1494b18b3f08254ae49eb63409a9efcf0b375ba/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/8cd6e8a4e311fbf0f96f31959b7ab9675a018ac5681ff16df929b726f662eece/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/f80d2ebf693491ae7b53662ac225c0ab7a5c8da56e18788ccaca903235912952/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/9171945fcfef0711e968bb5eb7561478885c53d8925b9abd325e7cb96902b74a/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/db8c27aa2c3f239e5ee2b36e6a3e371a5be1f31e76b8acdcf469c55f54c3d199/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/c545b8d23af2c1cc21b3f7edb24365731544ab1337bbc1d7b33f20504cfddbfa/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/3c7e5c158767958148cd9e2fad7816c64d13fc354754968d23da9cc3ec9e2d13/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/8d7213c53afbadea149ec3c855e37bc20bd8e708a1faee6b9f083f4745f9acf7/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d8b3d59e6f3a7e9da122698ace1a36b0174b2f30ddb85b02235134439d7d404b/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/9f28a071a8381733debabe18566f27e61c969e886b09a161f412188f36a1e302/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/878f5b32c601d4ad0af9313f217c3481dc195b1ffaf1f4d5235ffc29e7bf74c1/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/ff19f7944abf3645b8871a1de873fdab24dbce5e091173a84aeb61b88bfa81a4/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/b8bc48f45f8d95294d734572029ef6c44d8cdcaeba5ce1591d353c6bdb03c7e7/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/95e038e1eb331ca07f97ced184792532a669b37855604fdd3bab62009c923a17/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/9ae998882e40e13e1469bbd7fda014c714c621fa395fe1ba28916191180484e3/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/b3905f917fed3572acbd9d60ae34886efa2ea9f03fd76adadb89d48025576e43/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/cac765d39cc93f3f96db338fb8346dad9ee200f11483f7a05d937d64bc055a9e/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/9f24b918d96a84881d8a8f6a0fd439086d39f278c68ec72c5fa63b122ac724bb/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/f986912b4ee2adaba3b5ede94ca51f853dadafb316b863a2498f104b8b189bc2/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/f2bdca851eaa0da25281484fc6c19a6aefd247f519858f35f877fe0cb5ee4612/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/f12b60e980ca758a0268649c6f054de5484c6a27dfba4c74a99eee441d1e3a26/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/4132b54ba97bba02b6b17b3603b4edabf20da86176d286c098574b5ec2492349/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/7141cdb26d4299e6cc3c7f2aa4a31aa11ab1430ba14274864870a4b95aeb66ac/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/053c731e7b321f46da37c79f28182971c8cd92b7d31600e3f9181fadf9e178ef/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/4704121083a3f36f21554c2872bdda9a45c00bd4114bcf79c321ad07cf9eade7/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/666128f23b1f44a64f28391bee9e6a2ce3f5e3d2e8540cd6d51a6da608ffd6dc/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/bb498365d49da324c09656d7c74307a905f7c6b9a7404f8af12832a7f0ebd20d/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/f564f6abea4b6144a74d240dbd4a85b749866531662db6add4ea2a9b53cd178c/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/fb5be087ce85400ae4a2d5dec83463a533fcd8368d937d4998c718413b0d59e1/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/dc8e2ddd0673541d36081054de84219d30ba12a0b84b77526a071d4d8c38d5d7/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/b5f1ca2ce29308fb3d88afeab3718f157103f64933265e77cb971271cfa4f758/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/ee5d3ce6ac9b1cc771b028af49df110f1ab79df9fdb7dc9600cd0dddba771c4c/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/b0d3329171f7bf3f89af8aa83e73a21ba84c8740e26f120c29594a028d504a3b/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/199cdf9cbe36ffd6b142b16fe9234bcf47e7498ef1715715efd9f8dfe6758ea7/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/aa5a120630e095f1c05d857e58c22b7870fbfdcd4a37ad0cbff6d1b2d45eac60/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/9ce2df4b7b684802187816c8e6eb2ced57af24fa010564a360cb61521f3035b8/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/0874ec6bf6177489781edae27f01a8215f435d01c86c361c3d8f715e4d2f2a3f/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/80a18dcbb8a56f5eb0a9875d03219063f6fc08c914317308f56578abd90fd2f1/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/9bdd824d4cc59a6301227b77b154d3d4e8dc865c4d7e009413733fc77aeb17e7/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/e5d095b587bbffea126058393761f0fa5096e64931ec4b300d679065f6f46867/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/570008f1f9b7a6e426311d703957aef7045be733466706d5e83b67418cdf77fb/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/e48d998353d429ef56ee255e694666835c757a17576d3628c33cb6aff578faa1/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/a5dbd2eea502ef06914661c7e008cae81502f51fef247a4637e4a20ffbe42d73/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/6a4b85e183a5b09f00323d66a95a33791d148576c815e30c250d154029de8141/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/1f37aa54af92e22550d7691c57f2571f84cd37bc328c1648419a06a10b40178b/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/f487314236894cd9afa859c012f670f4a7ae5c8a6c57907ee30d5c9bd5e6dcc3/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/7f48ed1770c5ddcf1f7aea7932279855116949f69ce22aa07e4887de7a9306cc/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/d63e549036f6c81e82f8cd5ad3746bfa10c27a5abd7f9694a9015a82b7fc71e5/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/46dc9fa58a18e2e3f29be2a2d104afc29e296fb3bf0c4557be753a87639b3094/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/60f2c34791e14c8cb96999110690c334edd1f3900a895633c131d09a2755f54e/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/a8847c1a7fe699e3ca02a80f7625bf8a8aeefcbf0ba6e721aecbd5c3540fe602/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-502:{mountpoint:/var/lib/containers/storage/overlay/3ecd46d6ee935578ebef0964b2097d5acc1c458c8c4084ccb52d6ccdde06cc99/merged major:0 minor:502 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/392b409640b9b6027b26e3f3c36d663cce1303f31c2afd4401dcee62102e1541/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/48897b4c5f027af9b2f02d7809614de4324926efc24dc22e0eeec52825cfbed3/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/595aa1c4b6d6592f1317ec74ee8eb1b483bcf071719fca209c6d6d3da230743b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a804fa4fd7129e56a7c781142dd0e0f88a5f8ffdab18ea3f6b3572dc8bd7dde9/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/2a74a65614fcc8724381e2e4914aef04fb502c2f34a707a531c1bdb214a76017/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/4001d9debad8f239077f9d4354febf2ff4f60421aa41fd76d76225242efdd42e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/2f894a1a6c092714d4724bade1119a7c594f28eeb4690d3fe126a27a4028d581/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/c79bdccf57e9c5d87b0f9db3ace3e99f88a014ae5e58c8b8cc1e38f4c5140834/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c80c0bf31ac96806e017e9d43ee063f0d4c639a3e8161c9262be1d5f99fba77b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/71dfa5600139e997c5619c3588b9e17ca033dbf4af3bbc94cf09dc86c7fcdff3/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 19:29:01.725595 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.725053 5850 manager.go:217] Machine: {Timestamp:2026-11-02 19:29:01.72457565 +0000 UTC m=+0.066964723 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100179 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c42087a1b2f0cf32e4527c1c13b2 SystemUUID:ec26c420-87a1-b2f0-cf32-e4527c1c13b2 BootID:134be83f-6f7c-42ff-9df4-f9a419ff198f Filesystems:[{Device:/var/lib/kubelet/pods/405d3ed6-8332-46f0-b97d-326543807323/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f/volumes/kubernetes.io~projected/kube-api-access-28kv4 DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88f4fa50454796aaf822bc3612d51da1c57f8ba5155b13a1e488fd4da2a95df1/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43a8f63a-57a0-4be5-b55f-621b67342950/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43a8f63a-57a0-4be5-b55f-621b67342950/volumes/kubernetes.io~projected/kube-api-access-bgsrp DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:327 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~projected/kube-api-access-5nhrj DeviceMajor:0 DeviceMinor:338 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13da2219-8d3c-4b28-b6b0-38b51415de22/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:351 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19ec87c5-1d05-4829-9a6d-cddcd0f84607/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:352 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/125e3cca30e0c0415432b25dca84e50a35329edce80ff1b0b3d0ddc3690bf18b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/405d3ed6-8332-46f0-b97d-326543807323/volumes/kubernetes.io~projected/kube-api-access-q6j95 DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2/volumes/kubernetes.io~projected/kube-api-access-54czs DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:490 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/488c6e0df5e686add29118b2f232ca1d1e0f5e218f2361ce793f0b0934bde8b2/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f5bd0153977ea80cbdeebb54fbaafe9846cf48385a69ea2c02d3dd89d04e4ec/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f54c0ecffa913bcc377365538a402de44e285bd8b48c5aa16707496b8cb3a02/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c967790c-1a44-4339-aa6e-8becad9f4474/volumes/kubernetes.io~projected/kube-api-access-fjqwf DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/daf0d624-3b0c-4ceb-897a-c9d8b54213e2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a20ef282fdff3c3ed1d6f65c8c05a232f02e501e0c94acb02986b68934a92f2/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/884442753f0436d3d0b3c66bf30628040932f27f63cc660173957f4be66631ae/userdata/shm DeviceMajor:0 DeviceMinor:345 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ba7edd-403e-423f-b9a6-92c08b7881a9/volumes/kubernetes.io~projected/kube-api-access-7rkdj DeviceMajor:0 DeviceMinor:369 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d9559a8-3f33-4ea2-b478-8b9474d99ed7/volumes/kubernetes.io~projected/kube-api-access-d5bhk DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7acda1fa-9e95-4357-ba7b-7315edb4352e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/436a8b4fe9ec3b4600bfa808f8f091d66e6f712709acf0841b833d5c3d55fc79/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/770569fc3162f6a8fa8c1dc5e649325d3108c12182d41839520c33c02428e386/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd2a4d606f0a0c4df49062b4adcf1ff50aa7c43406c4a7c6b0ff5e023a05051a/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7acda1fa-9e95-4357-ba7b-7315edb4352e/volumes/kubernetes.io~projected/kube-api-access-qcgb7 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~projected/kube-api-access-kjvdn DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0c475c8b9e175d9934ce73ce681b18a11ba3f321f88afa49ec6a1a196980a9a/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42a08de9-3178-4d75-87b7-0cea7a813dd3/volumes/kubernetes.io~projected/kube-api-access-tq5p9 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:330 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:374 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48eccb46-4917-49bb-b8f0-f40cf28de5f6/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d733afefb0412f2afa49be26fbc7502b827d31df450cccad406a2c401b73a9a/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43a8f63a-57a0-4be5-b55f-621b67342950/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:281 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:373 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes/kubernetes.io~projected/kube-api-access-5zj4k DeviceMajor:0 DeviceMinor:496 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d9a6ad973d3cc9f414d9443c94cbd1984d66af12b8401a86ade6ca6dd2388b8/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/daf0d624-3b0c-4ceb-897a-c9d8b54213e2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:280 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~projected/kube-api-access-bpzm9 DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:326 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/107e58b3-1281-416a-b713-8efc20576e9f/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c00e59c137866958842ee52e3b2741e2f170e25f718ad32b342a18ab108add41/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d71e26dd-fe22-44a6-9b04-bf30e4aef71a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f31b231b-c403-428b-a1a3-f516644ac9e5/volumes/kubernetes.io~projected/kube-api-access-sb6f4 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb59f3c761721d25c75ceac6d48db30c50ac93b0fe5361d3b199e33e313c4799/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89590aeaf5245d153f1bbefae4e3f9adb43192d4db0960ddcd5a7d3ec0910213/userdata/shm DeviceMajor:0 DeviceMinor:394 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6446e2aebac2a76a71da81fe3d07269e7e2b1f4deef71b0608e84f70b5c6f130/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~projected/kube-api-access-zsz7l DeviceMajor:0 DeviceMinor:380 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6825ab200ababe5c3a84c253175e259e494c1f731647bf9841faef4a8f98c790/userdata/shm DeviceMajor:0 DeviceMinor:497 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-502 DeviceMajor:0 DeviceMinor:502 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47a581c4ec21a6c6ed42132dfa90f4296f0e49de45b8537352e115e71a010a86/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2867a688-7f22-4419-a69c-aef174888edc/volumes/kubernetes.io~projected/kube-api-access-dl8kl DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c15b0ba630be0437aaa4115339eb08210c93f67fbe02732fa4edfd6156558d17/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2867a688-7f22-4419-a69c-aef174888edc/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:263 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de1657c9-1cb9-44bb-b505-adf726ab2540/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2867a688-7f22-4419-a69c-aef174888edc/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0180ace-54a9-4796-a2ab-f069f965573c/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:376 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/daf0d624-3b0c-4ceb-897a-c9d8b54213e2/volumes/kubernetes.io~projected/kube-api-access-r6828 DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2602c042-a534-4767-8bb3-e5d2b93b3a62/volumes/kubernetes.io~projected/kube-api-access-vsg8n DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd10fc894d0a3082e0098d4444e6cd275aa026871f299331bcc904c7ccd6ea53/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2de0d2d-3838-4979-b2ec-c3ed7df1beb9/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2573d463444e98c321960ffca385dbadae267636744405fc811321d91d50db65/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/888426523d429d534a23b252e334071e1313754d18713d8f3dbaf7658ddee31c/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48fe43dd697504c22f3b846239e421a443aac2854ece9de26c4c8661810f20e6/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dfcdfe17db9bc3ac0a2af09c527fb7c32bdf8b17eb49aad1719f1e8638a3a8a/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/107e58b3-1281-416a-b713-8efc20576e9f/volumes/kubernetes.io~projected/kube-api-access-dvb8m DeviceMajor:0 DeviceMinor:344 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53fc9047-2f91-4806-acfd-d23b1b0cbc71/volumes/kubernetes.io~projected/kube-api-access-s6f7f DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97/volumes/kubernetes.io~projected/kube-api-access-hhdz2 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/010fac1e5b127921d19fae7d389c4501ef76dcada819b5030e50cfed167af910/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bfc75af335b31db6f21ecab7e09792756a3e511287d837b8275c12a1c04788d3/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:495 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19ec87c5-1d05-4829-9a6d-cddcd0f84607/volumes/kubernetes.io~projected/kube-api-access-n9lvp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24100d48-fca9-4735-81a6-a2455bf00804/volumes/kubernetes.io~projected/kube-api-access-5z4dh DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0f5bd0153977ea8 MacAddress:be:60:f5:19:bc:63 Speed:10000 Mtu:8901} {Name:2f54c0ecffa913b MacAddress:02:e9:a2:e2:d7:44 Speed:10000 Mtu:8901} {Name:436a8b4fe9ec3b4 MacAddress:02:12:51:35:d3:78 Speed:10000 Mtu:8901} {Name:488c6e0df5e686a MacAddress:be:6d:58:74:23:90 Speed:10000 Mtu:8901} {Name:6446e2aebac2a76 MacAddress:62:53:02:69:fb:12 Speed:10000 Mtu:8901} {Name:6825ab200ababe5 MacAddress:ea:e3:c9:04:20:47 Speed:10000 Mtu:8901} {Name:6a20ef282fdff3c MacAddress:9a:04:fa:20:ff:cf Speed:10000 Mtu:8901} {Name:770569fc3162f6a MacAddress:92:d9:65:c8:0f:8e Speed:10000 Mtu:8901} {Name:884442753f0436d MacAddress:e2:9d:db:3a:cf:e8 Speed:10000 Mtu:8901} {Name:888426523d429d5 MacAddress:1e:99:0b:8e:2d:1c Speed:10000 Mtu:8901} {Name:89590aeaf5245d1 MacAddress:62:c9:85:e5:c1:35 Speed:10000 Mtu:8901} {Name:8dfcdfe17db9bc3 MacAddress:2e:16:6c:33:17:cd Speed:10000 Mtu:8901} {Name:b0c475c8b9e175d MacAddress:ae:6c:91:df:6c:b8 Speed:10000 Mtu:8901} {Name:bfc75af335b31db MacAddress:86:49:79:24:d6:b5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e0:b0:3d:2b Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:1b:ab:3d:7f:07 Speed:0 Mtu:8901} {Name:c00e59c13786695 MacAddress:92:7c:73:62:8a:e1 Speed:10000 Mtu:8901} {Name:c15b0ba630be043 MacAddress:8e:3a:12:cc:78:04 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e0:b0:3d:2b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:52:52:27:ac:8f:17 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:06:0b:5c:c6:c4:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:29:01.725863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.725600 5850 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:29:01.725863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.725654 5850 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:29:01.725983 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.725956 5850 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:29:01.726127 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.725984 5850 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-244.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:29:01.726168 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726130 5850 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:29:01.726168 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726138 5850 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:29:01.726168 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726149 5850 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:29:01.726168 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726162 5850 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:29:01.726277 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726269 5850 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:29:01.726460 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726450 5850 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:29:01.726691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726592 5850 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:29:01.726691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726625 5850 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:29:01.726691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726638 5850 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:29:01.726691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726649 5850 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:29:01.726691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726659 5850 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:29:01.726691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.726667 5850 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:29:01.727377 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.727355 5850 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:29:01.727377 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.727370 5850 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:29:01.728008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.727990 5850 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:29:01.750120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750097 5850 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:29:01.750120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750114 5850 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:29:01.750440 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750252 5850 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 19:29:01.750520 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750500 5850 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:29:01.750556 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750531 5850 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:29:01.750661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750651 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:29:01.750693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750666 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:29:01.750693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750673 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:29:01.750693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750680 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:29:01.750693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750690 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:29:01.750864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750700 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:29:01.750864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750708 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:29:01.750864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750730 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:29:01.750864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750738 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:29:01.750864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750745 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:29:01.750864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750754 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:29:01.750864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750765 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:29:01.750864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750794 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:29:01.750864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.750802 5850 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:29:01.755744 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.755179 5850 server.go:1263] "Started kubelet" Nov 02 19:29:01.755744 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.755281 5850 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:29:01.755744 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.755456 5850 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:29:01.755744 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.755564 5850 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:29:01.755791 ip-10-0-131-244 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:29:01.756682 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.756657 5850 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:29:01.759013 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.758987 5850 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:29:01.766534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.765460 5850 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:29:01.766534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.765494 5850 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:29:01.766534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.765552 5850 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:22:49 +0000 UTC" deadline="2028-06-28 03:25:19.022119577 +0000 UTC" Nov 02 19:29:01.766534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.765589 5850 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14479h56m17.256533456s" Nov 02 19:29:01.766534 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:01.765601 5850 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-244.us-west-2.compute.internal\" not found" Nov 02 19:29:01.766534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.765649 5850 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:29:01.766534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.765656 5850 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:29:01.766534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.765715 5850 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:29:01.768501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.767466 5850 factory.go:55] Registering systemd factory Nov 02 19:29:01.768501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.767508 5850 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:29:01.768501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.767750 5850 factory.go:153] Registering CRI-O factory Nov 02 19:29:01.768501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.767770 5850 factory.go:223] Registration of the crio container factory successfully Nov 02 19:29:01.768501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.767832 5850 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:29:01.768501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.767851 5850 factory.go:103] Registering Raw factory Nov 02 19:29:01.768501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.767867 5850 manager.go:1199] Started watching for new ooms in manager Nov 02 19:29:01.768501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.768252 5850 manager.go:319] Starting recovery of all containers Nov 02 19:29:01.769196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769142 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9" seLinuxMountContext="" Nov 02 19:29:01.769244 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769208 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2752b72d-0a38-4fdd-ab0f-905bae22cedf" volumeName="kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config" seLinuxMountContext="" Nov 02 19:29:01.769244 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769225 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2867a688-7f22-4419-a69c-aef174888edc" volumeName="kubernetes.io/projected/2867a688-7f22-4419-a69c-aef174888edc-kube-api-access-dl8kl" seLinuxMountContext="" Nov 02 19:29:01.769244 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769238 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" volumeName="kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 19:29:01.769334 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769260 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1657c9-1cb9-44bb-b505-adf726ab2540" volumeName="kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 19:29:01.769334 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769293 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0180ace-54a9-4796-a2ab-f069f965573c" volumeName="kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client" seLinuxMountContext="" Nov 02 19:29:01.769334 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769306 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2867a688-7f22-4419-a69c-aef174888edc" volumeName="kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:29:01.769412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769331 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43a8f63a-57a0-4be5-b55f-621b67342950" volumeName="kubernetes.io/empty-dir/43a8f63a-57a0-4be5-b55f-621b67342950-volume-directive-shadow" seLinuxMountContext="" Nov 02 19:29:01.769412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769347 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eccb46-4917-49bb-b8f0-f40cf28de5f6" volumeName="kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 19:29:01.769412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769369 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eccb46-4917-49bb-b8f0-f40cf28de5f6" volumeName="kubernetes.io/empty-dir/48eccb46-4917-49bb-b8f0-f40cf28de5f6-audit-log" seLinuxMountContext="" Nov 02 19:29:01.769412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769389 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53fc9047-2f91-4806-acfd-d23b1b0cbc71" volumeName="kubernetes.io/projected/53fc9047-2f91-4806-acfd-d23b1b0cbc71-kube-api-access-s6f7f" seLinuxMountContext="" Nov 02 19:29:01.769538 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769404 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc6d9c0b-33ed-4ea1-9533-4ce89780bb97" volumeName="kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-tmp" seLinuxMountContext="" Nov 02 19:29:01.769538 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769427 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1657c9-1cb9-44bb-b505-adf726ab2540" volumeName="kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 19:29:01.769538 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769496 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24100d48-fca9-4735-81a6-a2455bf00804" volumeName="kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-multus-daemon-config" seLinuxMountContext="" Nov 02 19:29:01.769538 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769523 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42a08de9-3178-4d75-87b7-0cea7a813dd3" volumeName="kubernetes.io/projected/42a08de9-3178-4d75-87b7-0cea7a813dd3-kube-api-access-tq5p9" seLinuxMountContext="" Nov 02 19:29:01.769652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769536 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43a8f63a-57a0-4be5-b55f-621b67342950" volumeName="kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:29:01.769652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769558 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f" volumeName="kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-config" seLinuxMountContext="" Nov 02 19:29:01.769652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769572 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" volumeName="kubernetes.io/projected/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-kube-api-access-d5bhk" seLinuxMountContext="" Nov 02 19:29:01.769652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769584 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2de0d2d-3838-4979-b2ec-c3ed7df1beb9" volumeName="kubernetes.io/secret/a2de0d2d-3838-4979-b2ec-c3ed7df1beb9-tls-certificates" seLinuxMountContext="" Nov 02 19:29:01.769652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769633 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config" seLinuxMountContext="" Nov 02 19:29:01.769652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769646 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc6d9c0b-33ed-4ea1-9533-4ce89780bb97" volumeName="kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-tuned" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769660 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24100d48-fca9-4735-81a6-a2455bf00804" volumeName="kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-cni-binary-copy" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769681 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2" volumeName="kubernetes.io/configmap/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-config-volume" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769693 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1657c9-1cb9-44bb-b505-adf726ab2540" volumeName="kubernetes.io/empty-dir/de1657c9-1cb9-44bb-b505-adf726ab2540-tmp" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769706 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1657c9-1cb9-44bb-b505-adf726ab2540" volumeName="kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769733 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0180ace-54a9-4796-a2ab-f069f965573c" volumeName="kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-metrics-client-ca" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769748 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0180ace-54a9-4796-a2ab-f069f965573c" volumeName="kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-federate-client-tls" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769764 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2752b72d-0a38-4fdd-ab0f-905bae22cedf" volumeName="kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769785 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eccb46-4917-49bb-b8f0-f40cf28de5f6" volumeName="kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-metrics-server-audit-profiles" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769798 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2" volumeName="kubernetes.io/empty-dir/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-tmp-dir" seLinuxMountContext="" Nov 02 19:29:01.769820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769810 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2" volumeName="kubernetes.io/projected/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-kube-api-access-54czs" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769824 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2" volumeName="kubernetes.io/secret/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-metrics-tls" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769837 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769850 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769862 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1657c9-1cb9-44bb-b505-adf726ab2540" volumeName="kubernetes.io/configmap/de1657c9-1cb9-44bb-b505-adf726ab2540-metrics-client-ca" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769875 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19ec87c5-1d05-4829-9a6d-cddcd0f84607" volumeName="kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769887 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2602c042-a534-4767-8bb3-e5d2b93b3a62" volumeName="kubernetes.io/configmap/2602c042-a534-4767-8bb3-e5d2b93b3a62-iptables-alerter-script" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769900 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43a8f63a-57a0-4be5-b55f-621b67342950" volumeName="kubernetes.io/projected/43a8f63a-57a0-4be5-b55f-621b67342950-kube-api-access-bgsrp" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769913 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7acda1fa-9e95-4357-ba7b-7315edb4352e" volumeName="kubernetes.io/projected/7acda1fa-9e95-4357-ba7b-7315edb4352e-kube-api-access-qcgb7" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769925 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1657c9-1cb9-44bb-b505-adf726ab2540" volumeName="kubernetes.io/projected/de1657c9-1cb9-44bb-b505-adf726ab2540-kube-api-access-5nhrj" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769938 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24100d48-fca9-4735-81a6-a2455bf00804" volumeName="kubernetes.io/projected/24100d48-fca9-4735-81a6-a2455bf00804-kube-api-access-5z4dh" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769953 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="405d3ed6-8332-46f0-b97d-326543807323" volumeName="kubernetes.io/empty-dir/405d3ed6-8332-46f0-b97d-326543807323-data-volume" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769965 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eccb46-4917-49bb-b8f0-f40cf28de5f6" volumeName="kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-client-ca-bundle" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769980 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ba7edd-403e-423f-b9a6-92c08b7881a9" volumeName="kubernetes.io/empty-dir/b5ba7edd-403e-423f-b9a6-92c08b7881a9-tmp" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.769993 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c967790c-1a44-4339-aa6e-8becad9f4474" volumeName="kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770005 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71e26dd-fe22-44a6-9b04-bf30e4aef71a" volumeName="kubernetes.io/configmap/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-konnectivity-ca" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770018 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0180ace-54a9-4796-a2ab-f069f965573c" volumeName="kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-client-tls" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770031 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31b231b-c403-428b-a1a3-f516644ac9e5" volumeName="kubernetes.io/projected/f31b231b-c403-428b-a1a3-f516644ac9e5-kube-api-access-sb6f4" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770044 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="107e58b3-1281-416a-b713-8efc20576e9f" volumeName="kubernetes.io/secret/107e58b3-1281-416a-b713-8efc20576e9f-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770059 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2752b72d-0a38-4fdd-ab0f-905bae22cedf" volumeName="kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770071 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2867a688-7f22-4419-a69c-aef174888edc" volumeName="kubernetes.io/configmap/2867a688-7f22-4419-a69c-aef174888edc-metrics-client-ca" seLinuxMountContext="" Nov 02 19:29:01.770081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770085 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43a8f63a-57a0-4be5-b55f-621b67342950" volumeName="kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-metrics-client-ca" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770098 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43a8f63a-57a0-4be5-b55f-621b67342950" volumeName="kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770112 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770125 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770139 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2752b72d-0a38-4fdd-ab0f-905bae22cedf" volumeName="kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770162 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="405d3ed6-8332-46f0-b97d-326543807323" volumeName="kubernetes.io/projected/405d3ed6-8332-46f0-b97d-326543807323-kube-api-access-q6j95" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770174 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7acda1fa-9e95-4357-ba7b-7315edb4352e" volumeName="kubernetes.io/secret/7acda1fa-9e95-4357-ba7b-7315edb4352e-cert" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770187 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f" volumeName="kubernetes.io/secret/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770201 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770213 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770226 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc6d9c0b-33ed-4ea1-9533-4ce89780bb97" volumeName="kubernetes.io/projected/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-kube-api-access-hhdz2" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770239 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0180ace-54a9-4796-a2ab-f069f965573c" volumeName="kubernetes.io/projected/e0180ace-54a9-4796-a2ab-f069f965573c-kube-api-access-zsz7l" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770251 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2602c042-a534-4767-8bb3-e5d2b93b3a62" volumeName="kubernetes.io/projected/2602c042-a534-4767-8bb3-e5d2b93b3a62-kube-api-access-vsg8n" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770263 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2752b72d-0a38-4fdd-ab0f-905bae22cedf" volumeName="kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770275 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53fc9047-2f91-4806-acfd-d23b1b0cbc71" volumeName="kubernetes.io/configmap/53fc9047-2f91-4806-acfd-d23b1b0cbc71-serviceca" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770287 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770300 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ba7edd-403e-423f-b9a6-92c08b7881a9" volumeName="kubernetes.io/projected/b5ba7edd-403e-423f-b9a6-92c08b7881a9-kube-api-access-7rkdj" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770310 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf0d624-3b0c-4ceb-897a-c9d8b54213e2" volumeName="kubernetes.io/empty-dir/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-textfile" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770322 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf0d624-3b0c-4ceb-897a-c9d8b54213e2" volumeName="kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770332 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1657c9-1cb9-44bb-b505-adf726ab2540" volumeName="kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-grpc-tls" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770343 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" volumeName="kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-binary-copy" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770354 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf0d624-3b0c-4ceb-897a-c9d8b54213e2" volumeName="kubernetes.io/projected/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-kube-api-access-r6828" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770367 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0180ace-54a9-4796-a2ab-f069f965573c" volumeName="kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770378 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31b231b-c403-428b-a1a3-f516644ac9e5" volumeName="kubernetes.io/empty-dir/f31b231b-c403-428b-a1a3-f516644ac9e5-tmp-dir" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770392 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43a8f63a-57a0-4be5-b55f-621b67342950" volumeName="kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770408 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" volumeName="kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770421 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770432 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf0d624-3b0c-4ceb-897a-c9d8b54213e2" volumeName="kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-metrics-client-ca" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770443 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0180ace-54a9-4796-a2ab-f069f965573c" volumeName="kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:29:01.770654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770457 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="107e58b3-1281-416a-b713-8efc20576e9f" volumeName="kubernetes.io/projected/107e58b3-1281-416a-b713-8efc20576e9f-kube-api-access-dvb8m" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770677 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19ec87c5-1d05-4829-9a6d-cddcd0f84607" volumeName="kubernetes.io/projected/19ec87c5-1d05-4829-9a6d-cddcd0f84607-kube-api-access-n9lvp" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770709 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2752b72d-0a38-4fdd-ab0f-905bae22cedf" volumeName="kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770723 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770736 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770752 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf0d624-3b0c-4ceb-897a-c9d8b54213e2" volumeName="kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770765 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1657c9-1cb9-44bb-b505-adf726ab2540" volumeName="kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770778 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0180ace-54a9-4796-a2ab-f069f965573c" volumeName="kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770791 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2867a688-7f22-4419-a69c-aef174888edc" volumeName="kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-tls" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770806 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eccb46-4917-49bb-b8f0-f40cf28de5f6" volumeName="kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-tls" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770821 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f" volumeName="kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-env-overrides" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770836 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f" volumeName="kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-script-lib" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770848 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71e26dd-fe22-44a6-9b04-bf30e4aef71a" volumeName="kubernetes.io/secret/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-agent-certs" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770861 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf0d624-3b0c-4ceb-897a-c9d8b54213e2" volumeName="kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-tls" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770875 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13da2219-8d3c-4b28-b6b0-38b51415de22" volumeName="kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770887 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="405d3ed6-8332-46f0-b97d-326543807323" volumeName="kubernetes.io/secret/405d3ed6-8332-46f0-b97d-326543807323-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770900 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eccb46-4917-49bb-b8f0-f40cf28de5f6" volumeName="kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770914 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4c68e9-066f-4de3-9657-27fa1831f327" volumeName="kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770927 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1657c9-1cb9-44bb-b505-adf726ab2540" volumeName="kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770939 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2752b72d-0a38-4fdd-ab0f-905bae22cedf" volumeName="kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770952 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="405d3ed6-8332-46f0-b97d-326543807323" volumeName="kubernetes.io/configmap/405d3ed6-8332-46f0-b97d-326543807323-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770966 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eccb46-4917-49bb-b8f0-f40cf28de5f6" volumeName="kubernetes.io/projected/48eccb46-4917-49bb-b8f0-f40cf28de5f6-kube-api-access-kjvdn" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770978 5850 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f" volumeName="kubernetes.io/projected/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-kube-api-access-28kv4" seLinuxMountContext="" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.770991 5850 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:29:01.771466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.771000 5850 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:29:01.804604 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.804551 5850 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:29:01.806827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.806801 5850 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:29:01.807094 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.807079 5850 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:29:01.807194 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.807111 5850 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:29:01.807194 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:01.807154 5850 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 19:29:01.817130 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.817092 5850 generic.go:356] "Generic (PLEG): container finished" podID="daf0d624-3b0c-4ceb-897a-c9d8b54213e2" containerID="939f95f1a375f7a9abf3dbb0025e4195d45dca7c2a0e6d1bffea6632f88ed1b0" exitCode=0 Nov 02 19:29:01.845371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.845283 5850 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="584262aa807d770a8c6b6a74558d77aef8fd22575d45d7ec1682250fe44018b4" exitCode=0 Nov 02 19:29:01.845371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.845321 5850 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="e87365c15027646a6c6f0687a0e77bc27983042c9f1ca52e8af0157bca515901" exitCode=0 Nov 02 19:29:01.845371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.845333 5850 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="c6008217d8979d8ef43be30d8d74635a59fb7bc00c1d2cff6a19dab7dc41837b" exitCode=0 Nov 02 19:29:01.845371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.845342 5850 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="6210a725dbe4960fcf103832e35aea8b01966d3cbb304a520c4ed19e40aaa242" exitCode=0 Nov 02 19:29:01.845371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.845350 5850 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="76afb620a7c4e40917c60ebeac7cb6284680d613289493713e2d5cb967e63d18" exitCode=0 Nov 02 19:29:01.845371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.845359 5850 generic.go:356] "Generic (PLEG): container finished" podID="9d9559a8-3f33-4ea2-b478-8b9474d99ed7" containerID="603a22711f09a86054bd7b3a3d56f59f794c5cf9980b11df535d6679ef6bb254" exitCode=0 Nov 02 19:29:01.865746 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:01.865718 5850 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-244.us-west-2.compute.internal\" not found" Nov 02 19:29:01.866815 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.866779 5850 generic.go:356] "Generic (PLEG): container finished" podID="1cd5b763ddc1b8b385e585752b67b52f" containerID="ba1a05b137db964becc0a5e982de2804e00b44d4a26d19ea265058b9b28ee3f7" exitCode=0 Nov 02 19:29:01.907229 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:01.907195 5850 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 19:29:01.910202 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.910161 5850 generic.go:356] "Generic (PLEG): container finished" podID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerID="e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af" exitCode=0 Nov 02 19:29:01.919217 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.919194 5850 manager.go:324] Recovery completed Nov 02 19:29:01.944033 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.943991 5850 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 19:29:01.944033 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.944011 5850 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:29:01.944218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.944133 5850 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:29:01.946619 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.946603 5850 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:29:01.948792 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.948777 5850 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:29:01.948853 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.948811 5850 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:29:01.948853 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.948823 5850 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:29:01.951962 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.951934 5850 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:29:01.951962 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.951951 5850 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:29:01.951962 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.951958 5850 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:29:01.952126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.951971 5850 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:29:01.952126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952115 5850 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 19:29:01.952195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952127 5850 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 19:29:01.952195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952157 5850 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 19:29:01.952195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952162 5850 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 19:29:01.952195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952169 5850 policy_none.go:50] "Start" Nov 02 19:29:01.952195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952182 5850 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:29:01.952195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952193 5850 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:29:01.952370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952337 5850 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 19:29:01.952370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952345 5850 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 19:29:01.952370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.952353 5850 policy_none.go:44] "Start" Nov 02 19:29:01.953053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.953035 5850 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:29:01.953132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.953071 5850 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:29:01.953132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.953084 5850 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:29:01.958255 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.957683 5850 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:29:01.963987 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.963970 5850 manager.go:341] "Starting Device Plugin manager" Nov 02 19:29:01.964098 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:01.964003 5850 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:29:01.964098 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.964015 5850 server.go:87] "Starting device plugin registration server" Nov 02 19:29:01.964340 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.964325 5850 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:29:01.964403 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.964340 5850 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:29:01.964468 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.964449 5850 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:29:01.964591 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.964560 5850 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:29:01.964634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.964605 5850 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:29:01.964634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.964611 5850 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:29:01.964741 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.964649 5850 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:29:01.96458984Z","UUID":"47a7dc5e-a865-4bcb-b46f-bdc3146e915c","Handler":null,"Name":"","Endpoint":""} Nov 02 19:29:01.965981 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.965965 5850 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:29:01.966058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:01.965988 5850 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:29:01.966633 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:01.966587 5850 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:29:01.966633 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:01.966630 5850 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-244.us-west-2.compute.internal\" not found" Nov 02 19:29:02.047223 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.047185 5850 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:29:02.047396 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.047194 5850 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:29:02.065311 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.065266 5850 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:29:02.066828 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.066805 5850 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:29:02.066942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.066849 5850 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:29:02.066942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.066864 5850 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-244.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:29:02.066942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.066925 5850 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.108185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.108073 5850 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal"] Nov 02 19:29:02.108870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.108828 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"620b74bb5da064bc73a13b120097bf7c","Type":"ContainerStarted","Data":"fba3809ff8c3b599dc404cea8b1d4291913afc20d7e9295d80d9fe2aeb2e7139"} Nov 02 19:29:02.108990 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.108874 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"620b74bb5da064bc73a13b120097bf7c","Type":"ContainerStarted","Data":"fb59f3c761721d25c75ceac6d48db30c50ac93b0fe5361d3b199e33e313c4799"} Nov 02 19:29:02.108990 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.108957 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"1cd5b763ddc1b8b385e585752b67b52f","Type":"ContainerStarted","Data":"c4ec7183a50381e75180d2f0abedb05b25943466ba8754c9e1bf05673f0c1983"} Nov 02 19:29:02.108990 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.108976 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"1cd5b763ddc1b8b385e585752b67b52f","Type":"ContainerDied","Data":"ba1a05b137db964becc0a5e982de2804e00b44d4a26d19ea265058b9b28ee3f7"} Nov 02 19:29:02.109138 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.108994 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" event={"ID":"1cd5b763ddc1b8b385e585752b67b52f","Type":"ContainerStarted","Data":"010fac1e5b127921d19fae7d389c4501ef76dcada819b5030e50cfed167af910"} Nov 02 19:29:02.109138 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.109007 5850 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.109138 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.108973 5850 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.173914 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.173878 5850 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:29:02.174077 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:02.173943 5850 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.174573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.174559 5850 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:29:02.174657 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:02.174594 5850 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.183630 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.183603 5850 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.183737 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.183675 5850 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.246105 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.246064 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.246105 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.246108 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.246314 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.246134 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/620b74bb5da064bc73a13b120097bf7c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"620b74bb5da064bc73a13b120097bf7c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.347148 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.347112 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.347332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.347151 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.347332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.347180 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/620b74bb5da064bc73a13b120097bf7c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"620b74bb5da064bc73a13b120097bf7c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.347332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.347218 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.347332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.347254 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd5b763ddc1b8b385e585752b67b52f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"1cd5b763ddc1b8b385e585752b67b52f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.347332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.347298 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/620b74bb5da064bc73a13b120097bf7c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal\" (UID: \"620b74bb5da064bc73a13b120097bf7c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.727589 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.727547 5850 apiserver.go:52] "Watching apiserver" Nov 02 19:29:02.848285 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.848255 5850 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:29:02.849176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.849145 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8","openshift-cluster-node-tuning-operator/tuned-flwqt","openshift-dns/node-resolver-zlvtc","openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw","kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal","openshift-monitoring/alertmanager-main-0","openshift-monitoring/thanos-querier-64c99bbd67-6qj9p","openshift-console/console-6cb5d7db54-whf7r","openshift-multus/multus-additional-cni-plugins-cgzbx","openshift-network-diagnostics/network-check-target-ssjs9","openshift-ovn-kubernetes/ovnkube-node-8sdfd","openshift-insights/insights-runtime-extractor-vngml","openshift-multus/network-metrics-daemon-tq4ks","openshift-ingress-canary/ingress-canary-knrmt","openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq","openshift-multus/multus-dkvbt","openshift-console/downloads-6dd789c8b7-ckpzv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal","openshift-monitoring/monitoring-plugin-77f4f76594-2hv87","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t","openshift-image-registry/node-ca-7bzn9","openshift-network-operator/iptables-alerter-jhx4c","openshift-dns/dns-default-dbcdb","openshift-monitoring/node-exporter-cswh5","openshift-monitoring/telemeter-client-57ddbd48bb-l5d62","openshift-monitoring/metrics-server-868bfbd57c-9kws8","kube-system/global-pull-secret-syncer-r8qkh","kube-system/konnectivity-agent-4wwnx"] Nov 02 19:29:02.867801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.867771 5850 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:29:02.892869 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.892834 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:02.906357 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906322 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:29:02.906689 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906660 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:29:02.906689 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906405 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:29:02.906689 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906452 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906688 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906568 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906583 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906608 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906621 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906630 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906350 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906765 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906549 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906850 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906861 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:29:02.906940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.906912 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:29:02.916852 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.916829 5850 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.916983 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.916968 5850 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.950463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.950421 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6j95\" (UniqueName: \"kubernetes.io/projected/405d3ed6-8332-46f0-b97d-326543807323-kube-api-access-q6j95\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:02.950463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.950460 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1657c9-1cb9-44bb-b505-adf726ab2540-metrics-client-ca\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.950733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.950515 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-agent-certs\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:29:02.950733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.950541 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-systemd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.950733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.950565 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-federate-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.950733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.950588 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-wtmp\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.950888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.950832 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovn-node-metrics-cert\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.950968 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.950938 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:02.951023 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.950983 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.951023 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951011 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-metrics-tls\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:29:02.951093 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951046 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2867a688-7f22-4419-a69c-aef174888edc-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:29:02.951093 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951082 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cnibin\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.951183 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951115 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.951183 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951162 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-var-lib-kubelet\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.951281 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951189 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2de0d2d-3838-4979-b2ec-c3ed7df1beb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mkf7t\" (UID: \"a2de0d2d-3838-4979-b2ec-c3ed7df1beb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:29:02.951281 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951219 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.951281 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951245 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.951281 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951262 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.951281 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951279 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-k8s-cni-cncf-io\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.951501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951302 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-node-log\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.951501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951334 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31b231b-c403-428b-a1a3-f516644ac9e5-tmp-dir\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:29:02.951501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951401 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:02.951501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951406 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31b231b-c403-428b-a1a3-f516644ac9e5-tmp-dir\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:29:02.951665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951513 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.951665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951545 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-cni-binary-copy\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.951665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951581 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-conf\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.951665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951583 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.951665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951603 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-run\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.951876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951694 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:29:02.951876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951760 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.951876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951792 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-cni-binary-copy\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.951876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951821 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.951876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951842 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysconfig\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.951876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951860 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.952078 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951884 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvb8m\" (UniqueName: \"kubernetes.io/projected/107e58b3-1281-416a-b713-8efc20576e9f-kube-api-access-dvb8m\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:29:02.952078 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.951995 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.952078 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952018 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54czs\" (UniqueName: \"kubernetes.io/projected/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-kube-api-access-54czs\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:29:02.952078 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952038 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-ovn\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.952078 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952057 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-log-socket\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.952078 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952071 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53fc9047-2f91-4806-acfd-d23b1b0cbc71-serviceca\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:29:02.952331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952112 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-metrics-server-audit-profiles\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.952331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952167 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.952331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952264 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z4dh\" (UniqueName: \"kubernetes.io/projected/24100d48-fca9-4735-81a6-a2455bf00804-kube-api-access-5z4dh\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.952331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952298 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.952331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952323 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.952577 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952366 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-kubelet\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.952577 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952395 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-systemd-units\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.952577 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952420 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:02.952577 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952452 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.952577 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952550 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28kv4\" (UniqueName: \"kubernetes.io/projected/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-kube-api-access-28kv4\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.952758 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952622 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-tmp\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.952758 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952678 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-konnectivity-ca\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:29:02.952758 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952704 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-system-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.952917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.952692 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-tmp\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.953156 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953129 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5bhk\" (UniqueName: \"kubernetes.io/projected/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-kube-api-access-d5bhk\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.953310 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953260 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-etc-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.953366 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953347 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-env-overrides\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.953521 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953500 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-konnectivity-ca\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:29:02.953569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953541 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.953648 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953622 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.953648 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953626 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-env-overrides\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.953797 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953670 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsz7l\" (UniqueName: \"kubernetes.io/projected/e0180ace-54a9-4796-a2ab-f069f965573c-kube-api-access-zsz7l\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.953797 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953736 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-sys\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.953887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953813 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-textfile\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.953935 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953901 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-config\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.953985 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.953969 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-textfile\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.954134 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954114 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:29:02.954193 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954151 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.954245 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954185 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-config\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.954245 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954212 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-tls\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.954357 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954257 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6f4\" (UniqueName: \"kubernetes.io/projected/f31b231b-c403-428b-a1a3-f516644ac9e5-kube-api-access-sb6f4\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:29:02.954357 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954297 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/405d3ed6-8332-46f0-b97d-326543807323-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:02.954357 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954330 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.954523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954358 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-etc-kubernetes\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.954523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954393 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.954523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954427 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.954523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954462 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.954717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954514 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhrj\" (UniqueName: \"kubernetes.io/projected/de1657c9-1cb9-44bb-b505-adf726ab2540-kube-api-access-5nhrj\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.954717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954548 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6828\" (UniqueName: \"kubernetes.io/projected/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-kube-api-access-r6828\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.954717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954573 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6f7f\" (UniqueName: \"kubernetes.io/projected/53fc9047-2f91-4806-acfd-d23b1b0cbc71-kube-api-access-s6f7f\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:29:02.954717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954609 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/405d3ed6-8332-46f0-b97d-326543807323-data-volume\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:02.954717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954661 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:02.954717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954693 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.954717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954725 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2602c042-a534-4767-8bb3-e5d2b93b3a62-iptables-alerter-script\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:29:02.955028 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954759 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-root\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.955028 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954783 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.955028 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954791 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:02.955028 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.954930 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/405d3ed6-8332-46f0-b97d-326543807323-data-volume\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:02.955348 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955328 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-conf-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.955402 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955372 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-tuned\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.955450 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955407 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/405d3ed6-8332-46f0-b97d-326543807323-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:02.955450 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955435 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.955570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955448 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2602c042-a534-4767-8bb3-e5d2b93b3a62-iptables-alerter-script\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:29:02.955570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955468 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-socket-dir-parent\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.955570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955519 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-multus\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.955570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955552 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:29:02.955785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955587 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcgb7\" (UniqueName: \"kubernetes.io/projected/7acda1fa-9e95-4357-ba7b-7315edb4352e-kube-api-access-qcgb7\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:29:02.955785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955614 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-serving-certs-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.955785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955646 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.955785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955677 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-kubelet-config\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:29:02.955785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955709 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/43a8f63a-57a0-4be5-b55f-621b67342950-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:02.955785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955743 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.956021 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955810 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.956021 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.955930 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-metrics-client-ca\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.956107 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956044 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53fc9047-2f91-4806-acfd-d23b1b0cbc71-host\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:29:02.956107 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956053 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/43a8f63a-57a0-4be5-b55f-621b67342950-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:02.956107 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956091 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-kubelet\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.956410 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956391 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-tuned\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.956536 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956520 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-hostroot\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.956586 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956563 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-script-lib\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.956649 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956634 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-registration-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:02.956693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956671 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-device-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:02.956749 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956699 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-netns\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.956749 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956730 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-var-lib-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.956839 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956759 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-lib-modules\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.956839 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956785 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/405d3ed6-8332-46f0-b97d-326543807323-crio-socket\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:02.956921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956860 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.956921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956894 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvdn\" (UniqueName: \"kubernetes.io/projected/48eccb46-4917-49bb-b8f0-f40cf28de5f6-kube-api-access-kjvdn\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.957009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956925 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.957009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956959 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.957009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.956991 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-tmp-dir\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:29:02.957129 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957017 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.957129 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957064 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-systemd\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.957129 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957100 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f31b231b-c403-428b-a1a3-f516644ac9e5-hosts-file\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:29:02.957254 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957130 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.957254 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957161 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-bin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.957254 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957193 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-config-volume\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:29:02.957381 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957372 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-tls\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.957425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957401 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.957484 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957436 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-modprobe-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.957532 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957468 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-kubernetes\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.957532 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957516 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-sys\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.957624 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957551 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9lvp\" (UniqueName: \"kubernetes.io/projected/19ec87c5-1d05-4829-9a6d-cddcd0f84607-kube-api-access-n9lvp\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:29:02.957624 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957580 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de1657c9-1cb9-44bb-b505-adf726ab2540-tmp\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.957624 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957613 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:29:02.957748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957645 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-grpc-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.957748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957677 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-client-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.957748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957703 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-netns\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.957748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957736 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.957917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957771 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:02.957917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957803 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.957917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957836 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkdj\" (UniqueName: \"kubernetes.io/projected/b5ba7edd-403e-423f-b9a6-92c08b7881a9-kube-api-access-7rkdj\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:29:02.957917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957865 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:29:02.957917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957897 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-bin\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.958280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957929 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdz2\" (UniqueName: \"kubernetes.io/projected/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-kube-api-access-hhdz2\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.958280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957961 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-socket-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:02.958280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.957993 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsrp\" (UniqueName: \"kubernetes.io/projected/43a8f63a-57a0-4be5-b55f-621b67342950-kube-api-access-bgsrp\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:02.958280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.958022 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzm9\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.958280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.958053 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ba7edd-403e-423f-b9a6-92c08b7881a9-tmp\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:29:02.958280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.958088 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsg8n\" (UniqueName: \"kubernetes.io/projected/2602c042-a534-4767-8bb3-e5d2b93b3a62-kube-api-access-vsg8n\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:29:02.958681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.958346 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de1657c9-1cb9-44bb-b505-adf726ab2540-tmp\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.958681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.958352 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ba7edd-403e-423f-b9a6-92c08b7881a9-tmp\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:29:02.958843 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.958819 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.959846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.959819 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-sys-fs\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:02.959927 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.959848 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovnkube-script-lib\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.959927 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.959896 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.960043 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960020 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.960043 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960032 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-tmp-dir\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:29:02.960162 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960066 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48eccb46-4917-49bb-b8f0-f40cf28de5f6-audit-log\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.960162 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960097 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-cnibin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.960162 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960124 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-slash\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.960288 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960169 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.960288 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960179 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/48eccb46-4917-49bb-b8f0-f40cf28de5f6-audit-log\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.960288 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960212 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-host\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.960288 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960240 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-dbus\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:29:02.960288 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960268 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7acda1fa-9e95-4357-ba7b-7315edb4352e-cert\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:29:02.960543 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960312 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-multus-certs\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.960543 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960347 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8kl\" (UniqueName: \"kubernetes.io/projected/2867a688-7f22-4419-a69c-aef174888edc-kube-api-access-dl8kl\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:29:02.960543 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960375 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.960543 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960405 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-netd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.960543 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960442 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2602c042-a534-4767-8bb3-e5d2b93b3a62-host-slash\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:29:02.960543 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960495 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:02.960827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960544 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5p9\" (UniqueName: \"kubernetes.io/projected/42a08de9-3178-4d75-87b7-0cea7a813dd3-kube-api-access-tq5p9\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:02.960827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960587 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.960827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960624 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-client-certs\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.960827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960654 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zj4k\" (UniqueName: \"kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.960827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960681 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-multus-daemon-config\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:02.960827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960712 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-system-cni-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.960827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960759 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/107e58b3-1281-416a-b713-8efc20576e9f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:29:02.960827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960788 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-metrics-client-ca\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.960827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.960817 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.961095 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.961024 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:02.974084 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.974048 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 19:29:02.974084 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.974065 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fqq6c\"" Nov 02 19:29:02.974084 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.974077 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n2mjf\"" Nov 02 19:29:02.974318 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.974305 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dc6gd\"" Nov 02 19:29:02.974885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.974867 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 19:29:02.974885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.974880 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 19:29:02.974885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.974867 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mz7zj\"" Nov 02 19:29:02.975388 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975368 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:29:02.975458 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975371 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4nhx\"" Nov 02 19:29:02.975458 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975444 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9t85s\"" Nov 02 19:29:02.975577 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975496 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-c8nwr\"" Nov 02 19:29:02.975608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975586 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:29:02.975639 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975625 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 19:29:02.975680 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975638 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:29:02.975821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975809 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:29:02.975885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975844 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 19:29:02.975885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975862 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 19:29:02.975885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975862 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 19:29:02.976018 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.975949 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-52vlcnle3t3js\"" Nov 02 19:29:02.976205 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976178 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbvgr\"" Nov 02 19:29:02.976205 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976198 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:29:02.976205 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976206 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gsgfx\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976433 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976438 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njtg7\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976464 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976489 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976495 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976493 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c6zv8\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976543 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976550 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976553 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 19:29:02.976576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976573 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 19:29:02.976977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976813 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 19:29:02.976977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976814 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 19:29:02.976977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976866 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6bb8t\"" Nov 02 19:29:02.976977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976919 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wxdrr\"" Nov 02 19:29:02.976977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.976944 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fzxnx\"" Nov 02 19:29:02.977133 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977006 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 19:29:02.977133 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977027 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k9469\"" Nov 02 19:29:02.977133 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977045 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:29:02.977133 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977106 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 19:29:02.977133 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977107 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k9tfs\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977164 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977177 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977200 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-snhdz\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977213 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-4ltnd\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977228 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977250 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2gz6n\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977252 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sp8wk\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977270 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8h3sq919c5rg5\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977292 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977295 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4jh97\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977293 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977307 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977304 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977257 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4x665\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977314 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lnn7q\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977320 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 19:29:02.977332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977341 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bg56f\"" Nov 02 19:29:02.977864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977362 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:29:02.977864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977363 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:29:02.977864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977390 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:29:02.977864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977390 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 19:29:02.977864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977402 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nnqx8\"" Nov 02 19:29:02.977864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.977427 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-njjjh\"" Nov 02 19:29:02.978734 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.978719 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.978784 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.978770 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-grpc-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.979801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.979785 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.979890 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.979841 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-client-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.979921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.979875 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-tls\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.979960 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.979928 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:29:02.979960 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.979938 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2867a688-7f22-4419-a69c-aef174888edc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:29:02.980022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.979995 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.980160 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.980144 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19ec87c5-1d05-4829-9a6d-cddcd0f84607-metrics-certs\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:29:02.980251 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.980239 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.980285 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.980241 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.980874 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.980856 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.980874 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.980862 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-client-certs\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.981006 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.980857 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7acda1fa-9e95-4357-ba7b-7315edb4352e-cert\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:29:02.981006 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.980954 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-federate-client-tls\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.981006 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.980996 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d71e26dd-fe22-44a6-9b04-bf30e4aef71a-agent-certs\") pod \"konnectivity-agent-4wwnx\" (UID: \"d71e26dd-fe22-44a6-9b04-bf30e4aef71a\") " pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:29:02.981127 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.981033 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/107e58b3-1281-416a-b713-8efc20576e9f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:29:02.981185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.981171 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-ovn-node-metrics-cert\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:02.981239 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.981228 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:02.981310 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.981295 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-metrics-tls\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:29:02.981366 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.981339 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.981442 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.981423 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.981518 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.981497 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2de0d2d-3838-4979-b2ec-c3ed7df1beb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mkf7t\" (UID: \"a2de0d2d-3838-4979-b2ec-c3ed7df1beb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:29:02.981563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.981533 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.982205 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.982187 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13da2219-8d3c-4b28-b6b0-38b51415de22-original-pull-secret\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:29:02.983117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.983083 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:02.983117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.983099 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:02.983117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.983092 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.983274 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.983186 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.983384 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.983275 5850 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:29:02.983384 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:02.983310 5850 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.984298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.984278 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.984352 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.984341 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e0180ace-54a9-4796-a2ab-f069f965573c-secret-telemeter-client\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:02.984463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.984452 5850 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:29:02.984519 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:02.984504 5850 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-244.us-west-2.compute.internal" Nov 02 19:29:02.984849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.984834 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/48eccb46-4917-49bb-b8f0-f40cf28de5f6-secret-metrics-server-tls\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:02.985617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.985599 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:02.985659 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.985619 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.985687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.985657 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-tls\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.986806 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.986788 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de1657c9-1cb9-44bb-b505-adf726ab2540-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:02.986876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.986859 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:02.986910 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:02.986875 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/405d3ed6-8332-46f0-b97d-326543807323-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:03.006800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.006767 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:29:03.006800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.006782 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:29:03.006800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.006799 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:29:03.007097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.006835 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:29:03.007097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.006851 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 19:29:03.007097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.006878 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 19:29:03.007097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.006882 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:29:03.007097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.007021 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 19:29:03.007097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.007024 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 19:29:03.007942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.007922 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:29:03.007942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.007923 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 19:29:03.008091 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.007952 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 19:29:03.008091 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.008003 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 19:29:03.009174 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.009157 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-config-volume\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:29:03.011355 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.011337 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:03.012429 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.012409 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:03.012570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.012552 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/48eccb46-4917-49bb-b8f0-f40cf28de5f6-metrics-server-audit-profiles\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:03.013205 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.013190 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:29:03.013490 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.013457 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:29:03.013490 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.013466 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:29:03.013650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.013520 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:29:03.013650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.013524 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:29:03.013650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.013572 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:29:03.015964 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.015948 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:03.016129 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.016114 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-serving-certs-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:03.020086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.020061 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:29:03.020186 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.020088 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:29:03.020186 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.020103 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:29:03.020186 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.020068 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:29:03.021436 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.021417 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24100d48-fca9-4735-81a6-a2455bf00804-multus-daemon-config\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.022918 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.022893 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53fc9047-2f91-4806-acfd-d23b1b0cbc71-serviceca\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:29:03.025211 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.025194 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:29:03.025297 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.025281 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:29:03.025991 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.025975 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:29:03.026848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.026834 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 19:29:03.026848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.026837 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:29:03.026932 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.026850 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 19:29:03.028624 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.028605 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.029492 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.029584 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.029681 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5bhk\" (UniqueName: \"kubernetes.io/projected/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-kube-api-access-d5bhk\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.029682 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54czs\" (UniqueName: \"kubernetes.io/projected/9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2-kube-api-access-54czs\") pod \"dns-default-dbcdb\" (UID: \"9dbe0bd1-bc34-4664-9fa5-5d13952fe3d2\") " pod="openshift-dns/dns-default-dbcdb" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.029683 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28kv4\" (UniqueName: \"kubernetes.io/projected/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-kube-api-access-28kv4\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.029716 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.029715 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-metrics-client-ca\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.029872 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsz7l\" (UniqueName: \"kubernetes.io/projected/e0180ace-54a9-4796-a2ab-f069f965573c-kube-api-access-zsz7l\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030105 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvdn\" (UniqueName: \"kubernetes.io/projected/48eccb46-4917-49bb-b8f0-f40cf28de5f6-kube-api-access-kjvdn\") pod \"metrics-server-868bfbd57c-9kws8\" (UID: \"48eccb46-4917-49bb-b8f0-f40cf28de5f6\") " pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030216 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsg8n\" (UniqueName: \"kubernetes.io/projected/2602c042-a534-4767-8bb3-e5d2b93b3a62-kube-api-access-vsg8n\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030236 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6f4\" (UniqueName: \"kubernetes.io/projected/f31b231b-c403-428b-a1a3-f516644ac9e5-kube-api-access-sb6f4\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030290 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdz2\" (UniqueName: \"kubernetes.io/projected/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-kube-api-access-hhdz2\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030290 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvb8m\" (UniqueName: \"kubernetes.io/projected/107e58b3-1281-416a-b713-8efc20576e9f-kube-api-access-dvb8m\") pod \"monitoring-plugin-77f4f76594-2hv87\" (UID: \"107e58b3-1281-416a-b713-8efc20576e9f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030318 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6828\" (UniqueName: \"kubernetes.io/projected/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-kube-api-access-r6828\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030389 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030543 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgsrp\" (UniqueName: \"kubernetes.io/projected/43a8f63a-57a0-4be5-b55f-621b67342950-kube-api-access-bgsrp\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030815 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6j95\" (UniqueName: \"kubernetes.io/projected/405d3ed6-8332-46f0-b97d-326543807323-kube-api-access-q6j95\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030834 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z4dh\" (UniqueName: \"kubernetes.io/projected/24100d48-fca9-4735-81a6-a2455bf00804-kube-api-access-5z4dh\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.030942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030898 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6f7f\" (UniqueName: \"kubernetes.io/projected/53fc9047-2f91-4806-acfd-d23b1b0cbc71-kube-api-access-s6f7f\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.030987 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1657c9-1cb9-44bb-b505-adf726ab2540-metrics-client-ca\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.031031 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8kl\" (UniqueName: \"kubernetes.io/projected/2867a688-7f22-4419-a69c-aef174888edc-kube-api-access-dl8kl\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.031379 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcgb7\" (UniqueName: \"kubernetes.io/projected/7acda1fa-9e95-4357-ba7b-7315edb4352e-kube-api-access-qcgb7\") pod \"ingress-canary-knrmt\" (UID: \"7acda1fa-9e95-4357-ba7b-7315edb4352e\") " pod="openshift-ingress-canary/ingress-canary-knrmt" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.031598 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2867a688-7f22-4419-a69c-aef174888edc-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4rpfq\" (UID: \"2867a688-7f22-4419-a69c-aef174888edc\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4rpfq" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.031653 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/43a8f63a-57a0-4be5-b55f-621b67342950-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frwtw\" (UID: \"43a8f63a-57a0-4be5-b55f-621b67342950\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frwtw" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.031681 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.031776 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zj4k\" (UniqueName: \"kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.031787 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqwf\" (UniqueName: \"kubernetes.io/projected/c967790c-1a44-4339-aa6e-8becad9f4474-kube-api-access-fjqwf\") pod \"network-check-target-ssjs9\" (UID: \"c967790c-1a44-4339-aa6e-8becad9f4474\") " pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.031864 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5p9\" (UniqueName: \"kubernetes.io/projected/42a08de9-3178-4d75-87b7-0cea7a813dd3-kube-api-access-tq5p9\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.031978 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0180ace-54a9-4796-a2ab-f069f965573c-metrics-client-ca\") pod \"telemeter-client-57ddbd48bb-l5d62\" (UID: \"e0180ace-54a9-4796-a2ab-f069f965573c\") " pod="openshift-monitoring/telemeter-client-57ddbd48bb-l5d62" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.032042 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9lvp\" (UniqueName: \"kubernetes.io/projected/19ec87c5-1d05-4829-9a6d-cddcd0f84607-kube-api-access-n9lvp\") pod \"network-metrics-daemon-tq4ks\" (UID: \"19ec87c5-1d05-4829-9a6d-cddcd0f84607\") " pod="openshift-multus/network-metrics-daemon-tq4ks" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.032090 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzm9\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.032530 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhrj\" (UniqueName: \"kubernetes.io/projected/de1657c9-1cb9-44bb-b505-adf726ab2540-kube-api-access-5nhrj\") pod \"thanos-querier-64c99bbd67-6qj9p\" (UID: \"de1657c9-1cb9-44bb-b505-adf726ab2540\") " pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:03.032801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.032709 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkdj\" (UniqueName: \"kubernetes.io/projected/b5ba7edd-403e-423f-b9a6-92c08b7881a9-kube-api-access-7rkdj\") pod \"downloads-6dd789c8b7-ckpzv\" (UID: \"b5ba7edd-403e-423f-b9a6-92c08b7881a9\") " pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:29:03.035296 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.035273 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/405d3ed6-8332-46f0-b97d-326543807323-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:03.036444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.036429 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:29:03.036515 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.036433 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca\") pod \"console-6cb5d7db54-whf7r\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:03.061210 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061172 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cnibin\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:03.061210 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061216 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-var-lib-kubelet\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061240 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-k8s-cni-cncf-io\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061260 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-node-log\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061298 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-cnibin\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061311 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-var-lib-kubelet\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061328 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-conf\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061307 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-k8s-cni-cncf-io\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061307 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-node-log\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061351 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-run\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061382 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-run\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061398 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysconfig\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.061432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061426 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-ovn\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061451 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-log-socket\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061491 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-conf\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061500 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061516 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-ovn\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061526 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-kubelet\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061541 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-systemd-units\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061547 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-log-socket\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061556 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061557 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-kubelet\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061504 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysconfig\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061575 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061593 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-systemd-units\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061601 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-system-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061619 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061632 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-etc-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061655 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-sys\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061674 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-etc-kubernetes\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061682 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-system-cni-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061695 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-sys\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061701 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-root\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061713 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-etc-kubernetes\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061717 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061675 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-etc-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061743 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061722 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-root\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061753 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-conf-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061784 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-socket-dir-parent\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061807 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-multus\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061783 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-conf-dir\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061825 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-kubelet-config\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061843 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-multus\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061857 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-kubelet-config\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061864 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-multus-socket-dir-parent\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061874 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53fc9047-2f91-4806-acfd-d23b1b0cbc71-host\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061891 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-kubelet\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061909 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-hostroot\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.061903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061915 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53fc9047-2f91-4806-acfd-d23b1b0cbc71-host\") pod \"node-ca-7bzn9\" (UID: \"53fc9047-2f91-4806-acfd-d23b1b0cbc71\") " pod="openshift-image-registry/node-ca-7bzn9" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061931 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-registration-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061949 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-device-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061959 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-hostroot\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061961 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-kubelet\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061963 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-netns\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061986 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-registration-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.061994 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-netns\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062013 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-device-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062027 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-var-lib-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062058 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-lib-modules\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062079 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/405d3ed6-8332-46f0-b97d-326543807323-crio-socket\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062082 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-var-lib-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062098 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-systemd\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062126 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f31b231b-c403-428b-a1a3-f516644ac9e5-hosts-file\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062136 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/405d3ed6-8332-46f0-b97d-326543807323-crio-socket\") pod \"insights-runtime-extractor-vngml\" (UID: \"405d3ed6-8332-46f0-b97d-326543807323\") " pod="openshift-insights/insights-runtime-extractor-vngml" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062142 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-bin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062161 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-systemd\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062165 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-var-lib-cni-bin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062137 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-lib-modules\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062175 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-modprobe-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062201 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-kubernetes\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062208 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f31b231b-c403-428b-a1a3-f516644ac9e5-hosts-file\") pod \"node-resolver-zlvtc\" (UID: \"f31b231b-c403-428b-a1a3-f516644ac9e5\") " pod="openshift-dns/node-resolver-zlvtc" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062225 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-sys\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062251 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-kubernetes\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062252 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-netns\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062267 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-modprobe-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062282 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062296 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-sys\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062305 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-netns\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062310 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-bin\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062323 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-openvswitch\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062334 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-socket-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062362 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062411 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-bin\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062418 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062413 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-socket-dir\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062433 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-sys-fs\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062455 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-cnibin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062486 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-slash\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062506 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42a08de9-3178-4d75-87b7-0cea7a813dd3-sys-fs\") pod \"aws-ebs-csi-driver-node-c7ct8\" (UID: \"42a08de9-3178-4d75-87b7-0cea7a813dd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c7ct8" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062511 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062530 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-host\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062537 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062545 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-dbus\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062564 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-multus-certs\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062565 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-cnibin\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062580 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062584 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24100d48-fca9-4735-81a6-a2455bf00804-host-run-multus-certs\") pod \"multus-dkvbt\" (UID: \"24100d48-fca9-4735-81a6-a2455bf00804\") " pod="openshift-multus/multus-dkvbt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062565 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-slash\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062595 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-netd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062602 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13da2219-8d3c-4b28-b6b0-38b51415de22-dbus\") pod \"global-pull-secret-syncer-r8qkh\" (UID: \"13da2219-8d3c-4b28-b6b0-38b51415de22\") " pod="kube-system/global-pull-secret-syncer-r8qkh" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062614 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-cni-netd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062577 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-host\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062622 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2602c042-a534-4767-8bb3-e5d2b93b3a62-host-slash\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062642 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062661 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062669 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2602c042-a534-4767-8bb3-e5d2b93b3a62-host-slash\") pod \"iptables-alerter-jhx4c\" (UID: \"2602c042-a534-4767-8bb3-e5d2b93b3a62\") " pod="openshift-network-operator/iptables-alerter-jhx4c" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062688 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-system-cni-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062712 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-systemd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062746 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-wtmp\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062766 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc6d9c0b-33ed-4ea1-9533-4ce89780bb97-etc-sysctl-d\") pod \"tuned-flwqt\" (UID: \"dc6d9c0b-33ed-4ea1-9533-4ce89780bb97\") " pod="openshift-cluster-node-tuning-operator/tuned-flwqt" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062786 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f-run-systemd\") pod \"ovnkube-node-8sdfd\" (UID: \"84fc00c2-cfbb-4d44-867e-0b9b7c61cd0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062749 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d9559a8-3f33-4ea2-b478-8b9474d99ed7-system-cni-dir\") pod \"multus-additional-cni-plugins-cgzbx\" (UID: \"9d9559a8-3f33-4ea2-b478-8b9474d99ed7\") " pod="openshift-multus/multus-additional-cni-plugins-cgzbx" Nov 02 19:29:03.063233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.062864 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/daf0d624-3b0c-4ceb-897a-c9d8b54213e2-node-exporter-wtmp\") pod \"node-exporter-cswh5\" (UID: \"daf0d624-3b0c-4ceb-897a-c9d8b54213e2\") " pod="openshift-monitoring/node-exporter-cswh5" Nov 02 19:29:03.165689 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.165667 5850 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:29:03.619563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.619524 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:03.815054 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.815021 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:03.933286 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:03.933213 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:04.236025 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:04.235901 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:29:04.458158 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:04.458109 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:29:04.465041 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:04.463751 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hv87" Nov 02 19:29:05.193412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:05.193375 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbcdb" Nov 02 19:29:06.369946 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:06.369909 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4wwnx" Nov 02 19:29:06.738092 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:06.738007 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:06.743987 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:06.743947 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-64c99bbd67-6qj9p" Nov 02 19:29:06.834923 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:06.834894 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mkf7t" Nov 02 19:29:07.013552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:07.013445 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:07.013727 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:07.013713 5850 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:29:07.553153 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:07.553119 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:07.806989 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:07.806913 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:07.940939 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:07.940903 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb5d7db54-whf7r" event={"ID":"2752b72d-0a38-4fdd-ab0f-905bae22cedf","Type":"ContainerStarted","Data":"36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6"} Nov 02 19:29:07.954805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:07.954772 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:08.005114 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:08.005053 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:08.144254 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:08.144218 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:08.293259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:08.293226 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:08.527780 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:08.527704 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sdfd" Nov 02 19:29:10.605965 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:10.605877 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:11.231145 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.231110 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84f584d8bb-wsxmb"] Nov 02 19:29:11.234660 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.234637 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.344089 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.344050 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74918778-82ad-4bde-bdaf-7d913a496a04-tmp\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.344255 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.344095 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-oauth-config\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.344255 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.344220 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-oauth-serving-cert\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.344372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.344265 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l674d\" (UniqueName: \"kubernetes.io/projected/74918778-82ad-4bde-bdaf-7d913a496a04-kube-api-access-l674d\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.344372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.344342 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-console-config\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.344466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.344403 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-serving-cert\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.344466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.344433 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-trusted-ca-bundle\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.344596 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.344582 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-service-ca\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.444535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.444501 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 19:29:11.445111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445087 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74918778-82ad-4bde-bdaf-7d913a496a04-tmp\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.445193 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445121 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-oauth-config\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.445193 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445164 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-oauth-serving-cert\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.445294 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445202 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l674d\" (UniqueName: \"kubernetes.io/projected/74918778-82ad-4bde-bdaf-7d913a496a04-kube-api-access-l674d\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.445294 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445232 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-console-config\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.445294 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445274 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-serving-cert\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.445450 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445295 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-trusted-ca-bundle\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.445450 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445349 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-service-ca\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.445595 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445576 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74918778-82ad-4bde-bdaf-7d913a496a04-tmp\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.445931 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.445908 5850 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:29:11.446084 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.446066 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-service-ca\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.446177 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.446158 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f584d8bb-wsxmb"] Nov 02 19:29:11.446349 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.446326 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-oauth-serving-cert\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.446786 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.446735 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-console-config\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.446954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.446932 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-trusted-ca-bundle\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.448761 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.448739 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-oauth-config\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.448826 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.448761 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-serving-cert\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.504023 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.503957 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l674d\" (UniqueName: \"kubernetes.io/projected/74918778-82ad-4bde-bdaf-7d913a496a04-kube-api-access-l674d\") pod \"console-84f584d8bb-wsxmb\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.548975 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.548941 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:11.928964 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.928933 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f584d8bb-wsxmb"] Nov 02 19:29:11.929308 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:29:11.929216 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74918778_82ad_4bde_bdaf_7d913a496a04.slice/crio-8fcb6199b4db27b2b9a7b9ab6d75dbed53778f66a9e65f7613142024e58367d4 WatchSource:0}: Error finding container 8fcb6199b4db27b2b9a7b9ab6d75dbed53778f66a9e65f7613142024e58367d4: Status 404 returned error can't find the container with id 8fcb6199b4db27b2b9a7b9ab6d75dbed53778f66a9e65f7613142024e58367d4 Nov 02 19:29:11.961357 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:11.961318 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f584d8bb-wsxmb" event={"ID":"74918778-82ad-4bde-bdaf-7d913a496a04","Type":"ContainerStarted","Data":"8fcb6199b4db27b2b9a7b9ab6d75dbed53778f66a9e65f7613142024e58367d4"} Nov 02 19:29:12.966075 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:12.966026 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f584d8bb-wsxmb" event={"ID":"74918778-82ad-4bde-bdaf-7d913a496a04","Type":"ContainerStarted","Data":"787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353"} Nov 02 19:29:13.154143 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:13.154107 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:14.760916 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:14.760882 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:29:16.637244 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:16.637194 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:16.642970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:16.642939 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:16.761316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:16.761238 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84f584d8bb-wsxmb" podStartSLOduration=5.761224453 podStartE2EDuration="5.761224453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:29:13.135871444 +0000 UTC m=+11.478260522" watchObservedRunningTime="2026-11-02 19:29:16.761224453 +0000 UTC m=+15.103613532" Nov 02 19:29:16.988664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:16.988570 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:17.490198 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:17.490163 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:17.648317 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:17.648282 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:21.460309 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:21.460272 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:22.874836 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:22.874796 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:22.879641 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:22.879614 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:23.009701 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:23.009617 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-ckpzv" event={"ID":"b5ba7edd-403e-423f-b9a6-92c08b7881a9","Type":"ContainerStarted","Data":"c4f00d5425d9cfb5ae98a5fb102b5f9c8fb333bbae0979b45e2f73ec54bad568"} Nov 02 19:29:23.013978 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:23.013943 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:29:23.277994 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:23.277911 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cb5d7db54-whf7r"] Nov 02 19:29:25.197961 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:25.197922 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:25.363828 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:25.363792 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:27.558700 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:27.558668 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:27.563204 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:27.563174 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-868bfbd57c-9kws8" Nov 02 19:29:32.485068 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:32.485029 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:32.640934 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:32.640881 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:33.009998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:33.009963 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:29:33.011551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:33.011532 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-ckpzv" Nov 02 19:29:34.238617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:34.238587 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ssjs9" Nov 02 19:29:42.124570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:42.124532 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:42.277850 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:42.277815 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:44.760404 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:44.760364 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:29:47.531157 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:47.531120 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:47.692068 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:47.692035 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:48.304712 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.304667 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6cb5d7db54-whf7r" podUID="2752b72d-0a38-4fdd-ab0f-905bae22cedf" containerName="console" containerID="cri-o://36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6" gracePeriod=15 Nov 02 19:29:48.585900 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.585824 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:48.689398 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689355 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert\") pod \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " Nov 02 19:29:48.689398 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689405 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp\") pod \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " Nov 02 19:29:48.689673 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689442 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config\") pod \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " Nov 02 19:29:48.689673 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689465 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config\") pod \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " Nov 02 19:29:48.689673 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689514 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert\") pod \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " Nov 02 19:29:48.689673 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689558 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k\" (UniqueName: \"kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k\") pod \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " Nov 02 19:29:48.689673 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689586 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca\") pod \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\" (UID: \"2752b72d-0a38-4fdd-ab0f-905bae22cedf\") " Nov 02 19:29:48.689925 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689794 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp" pod "2752b72d-0a38-4fdd-ab0f-905bae22cedf" (UID: "2752b72d-0a38-4fdd-ab0f-905bae22cedf"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:29:48.690006 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689973 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert" pod "2752b72d-0a38-4fdd-ab0f-905bae22cedf" (UID: "2752b72d-0a38-4fdd-ab0f-905bae22cedf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:29:48.690057 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.689980 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config" pod "2752b72d-0a38-4fdd-ab0f-905bae22cedf" (UID: "2752b72d-0a38-4fdd-ab0f-905bae22cedf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:29:48.690057 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.690044 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca" pod "2752b72d-0a38-4fdd-ab0f-905bae22cedf" (UID: "2752b72d-0a38-4fdd-ab0f-905bae22cedf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:29:48.691992 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.691966 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k" pod "2752b72d-0a38-4fdd-ab0f-905bae22cedf" (UID: "2752b72d-0a38-4fdd-ab0f-905bae22cedf"). InnerVolumeSpecName "kube-api-access-5zj4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:29:48.692209 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.692186 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert" pod "2752b72d-0a38-4fdd-ab0f-905bae22cedf" (UID: "2752b72d-0a38-4fdd-ab0f-905bae22cedf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:29:48.692271 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.692238 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config" pod "2752b72d-0a38-4fdd-ab0f-905bae22cedf" (UID: "2752b72d-0a38-4fdd-ab0f-905bae22cedf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:29:48.790151 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.790108 5850 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-serving-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:48.790151 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.790142 5850 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2752b72d-0a38-4fdd-ab0f-905bae22cedf-tmp\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:48.790151 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.790153 5850 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-config\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:48.790151 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.790165 5850 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2752b72d-0a38-4fdd-ab0f-905bae22cedf-console-oauth-config\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:48.790463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.790175 5850 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-oauth-serving-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:48.790463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.790185 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zj4k\" (UniqueName: \"kubernetes.io/projected/2752b72d-0a38-4fdd-ab0f-905bae22cedf-kube-api-access-5zj4k\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:48.790463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:48.790194 5850 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2752b72d-0a38-4fdd-ab0f-905bae22cedf-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:49.110970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.110938 5850 generic.go:356] "Generic (PLEG): container finished" podID="2752b72d-0a38-4fdd-ab0f-905bae22cedf" containerID="36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6" exitCode=0 Nov 02 19:29:49.111141 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.110984 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb5d7db54-whf7r" Nov 02 19:29:49.111141 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.111023 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb5d7db54-whf7r" event={"ID":"2752b72d-0a38-4fdd-ab0f-905bae22cedf","Type":"ContainerDied","Data":"36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6"} Nov 02 19:29:49.111141 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.111079 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb5d7db54-whf7r" event={"ID":"2752b72d-0a38-4fdd-ab0f-905bae22cedf","Type":"ContainerDied","Data":"6825ab200ababe5c3a84c253175e259e494c1f731647bf9841faef4a8f98c790"} Nov 02 19:29:49.111141 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.111106 5850 scope.go:122] "RemoveContainer" containerID="36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6" Nov 02 19:29:49.119662 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.119643 5850 scope.go:122] "RemoveContainer" containerID="36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6" Nov 02 19:29:49.119958 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:29:49.119936 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6\": container with ID starting with 36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6 not found: ID does not exist" containerID="36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6" Nov 02 19:29:49.120053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.119964 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6"} err="failed to get container status \"36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6\": rpc error: code = NotFound desc = could not find container \"36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6\": container with ID starting with 36d585b791714ee849c0a3e11e0aab7e3a8774703d9fde04f91f9d96c3f991a6 not found: ID does not exist" Nov 02 19:29:49.278391 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.278356 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cb5d7db54-whf7r"] Nov 02 19:29:49.281814 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.281787 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cb5d7db54-whf7r"] Nov 02 19:29:49.812430 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:49.812399 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2752b72d-0a38-4fdd-ab0f-905bae22cedf" path="/var/lib/kubelet/pods/2752b72d-0a38-4fdd-ab0f-905bae22cedf/volumes" Nov 02 19:29:56.078672 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:56.078636 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:29:56.230791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:29:56.230745 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:03.395036 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:03.395001 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:30:03.395548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:03.395304 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="alertmanager" containerID="cri-o://069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9" gracePeriod=120 Nov 02 19:30:03.395548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:03.395378 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy-metric" containerID="cri-o://27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0" gracePeriod=120 Nov 02 19:30:03.395548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:03.395413 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy-web" containerID="cri-o://00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c" gracePeriod=120 Nov 02 19:30:03.395548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:03.395436 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="config-reloader" containerID="cri-o://1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a" gracePeriod=120 Nov 02 19:30:03.395548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:03.395401 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="prom-label-proxy" containerID="cri-o://80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1" gracePeriod=120 Nov 02 19:30:03.395548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:03.395492 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy" containerID="cri-o://d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d" gracePeriod=120 Nov 02 19:30:03.927723 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:03.927689 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:04.168718 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168681 5850 generic.go:356] "Generic (PLEG): container finished" podID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerID="80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1" exitCode=0 Nov 02 19:30:04.168718 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168706 5850 generic.go:356] "Generic (PLEG): container finished" podID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerID="27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0" exitCode=0 Nov 02 19:30:04.168718 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168713 5850 generic.go:356] "Generic (PLEG): container finished" podID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerID="d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d" exitCode=0 Nov 02 19:30:04.168718 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168722 5850 generic.go:356] "Generic (PLEG): container finished" podID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerID="1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a" exitCode=0 Nov 02 19:30:04.168718 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168728 5850 generic.go:356] "Generic (PLEG): container finished" podID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerID="069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9" exitCode=0 Nov 02 19:30:04.169009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168754 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerDied","Data":"80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1"} Nov 02 19:30:04.169009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168823 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerDied","Data":"27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0"} Nov 02 19:30:04.169009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168850 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerDied","Data":"d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d"} Nov 02 19:30:04.169009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168876 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerDied","Data":"1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a"} Nov 02 19:30:04.169009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.168899 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerDied","Data":"069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9"} Nov 02 19:30:04.261776 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.261693 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:04.675350 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.675312 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:04.724162 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724133 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724353 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724186 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724353 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724211 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724353 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724246 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724353 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724271 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724353 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724287 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724353 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724316 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724685 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724368 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724741 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724706 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:30:04.724870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724839 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724927 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724900 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.724982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724939 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.725040 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.724981 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.725040 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.725020 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets\") pod \"aa4c68e9-066f-4de3-9657-27fa1831f327\" (UID: \"aa4c68e9-066f-4de3-9657-27fa1831f327\") " Nov 02 19:30:04.725279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.725260 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-metrics-client-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.725390 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.725365 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:30:04.725848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.725779 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:30:04.727685 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.727645 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:30:04.727848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.727809 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:30:04.727953 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.727926 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:30:04.728098 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.728074 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:30:04.728220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.728174 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:30:04.728415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.728395 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "kube-api-access-bpzm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:30:04.728988 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.728967 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:30:04.729752 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.729723 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:30:04.732177 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.732153 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:30:04.740022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.739987 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config" pod "aa4c68e9-066f-4de3-9657-27fa1831f327" (UID: "aa4c68e9-066f-4de3-9657-27fa1831f327"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:30:04.826516 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826414 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpzm9\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-kube-api-access-bpzm9\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826516 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826448 5850 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa4c68e9-066f-4de3-9657-27fa1831f327-tls-assets\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826516 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826464 5850 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-main-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826516 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826519 5850 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-web-config\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826532 5850 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-config-out\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826546 5850 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826556 5850 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826566 5850 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-cluster-tls-config\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826576 5850 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826593 5850 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826603 5850 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa4c68e9-066f-4de3-9657-27fa1831f327-alertmanager-main-db\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:04.826756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:04.826618 5850 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa4c68e9-066f-4de3-9657-27fa1831f327-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:30:05.179004 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.178969 5850 generic.go:356] "Generic (PLEG): container finished" podID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerID="00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c" exitCode=0 Nov 02 19:30:05.179185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.179018 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerDied","Data":"00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c"} Nov 02 19:30:05.179185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.179057 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.179185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.179108 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa4c68e9-066f-4de3-9657-27fa1831f327","Type":"ContainerDied","Data":"8dfcdfe17db9bc3ac0a2af09c527fb7c32bdf8b17eb49aad1719f1e8638a3a8a"} Nov 02 19:30:05.179185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.179143 5850 scope.go:122] "RemoveContainer" containerID="80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1" Nov 02 19:30:05.189064 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.189037 5850 scope.go:122] "RemoveContainer" containerID="27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0" Nov 02 19:30:05.197432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.197406 5850 scope.go:122] "RemoveContainer" containerID="d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d" Nov 02 19:30:05.204972 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.204952 5850 scope.go:122] "RemoveContainer" containerID="00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c" Nov 02 19:30:05.212579 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.212549 5850 scope.go:122] "RemoveContainer" containerID="1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a" Nov 02 19:30:05.220537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.220507 5850 scope.go:122] "RemoveContainer" containerID="069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9" Nov 02 19:30:05.228384 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.228360 5850 scope.go:122] "RemoveContainer" containerID="e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af" Nov 02 19:30:05.236228 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.236202 5850 scope.go:122] "RemoveContainer" containerID="80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1" Nov 02 19:30:05.236562 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:30:05.236541 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1\": container with ID starting with 80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1 not found: ID does not exist" containerID="80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1" Nov 02 19:30:05.236636 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.236573 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1"} err="failed to get container status \"80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1\": rpc error: code = NotFound desc = could not find container \"80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1\": container with ID starting with 80ebae666ec5a950743ad2036feab1c4b4f491486ac7859528e79cd5ba515bc1 not found: ID does not exist" Nov 02 19:30:05.236636 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.236592 5850 scope.go:122] "RemoveContainer" containerID="27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0" Nov 02 19:30:05.236891 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:30:05.236874 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0\": container with ID starting with 27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0 not found: ID does not exist" containerID="27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0" Nov 02 19:30:05.236933 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.236895 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0"} err="failed to get container status \"27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0\": rpc error: code = NotFound desc = could not find container \"27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0\": container with ID starting with 27c3e4c16914a1252982d7403cdf5f5ad905284dd807dd2ae147b9e72693a5b0 not found: ID does not exist" Nov 02 19:30:05.236933 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.236908 5850 scope.go:122] "RemoveContainer" containerID="d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d" Nov 02 19:30:05.237110 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:30:05.237096 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d\": container with ID starting with d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d not found: ID does not exist" containerID="d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d" Nov 02 19:30:05.237145 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.237114 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d"} err="failed to get container status \"d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d\": rpc error: code = NotFound desc = could not find container \"d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d\": container with ID starting with d0d638f97c5f13f709ee0894a36490c51216825a56139f932cd4fa75ad242c8d not found: ID does not exist" Nov 02 19:30:05.237145 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.237126 5850 scope.go:122] "RemoveContainer" containerID="00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c" Nov 02 19:30:05.237338 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:30:05.237320 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c\": container with ID starting with 00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c not found: ID does not exist" containerID="00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c" Nov 02 19:30:05.237380 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.237341 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c"} err="failed to get container status \"00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c\": rpc error: code = NotFound desc = could not find container \"00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c\": container with ID starting with 00bc519d266290798a5ab1b43f5761d23e86e3b9db41ef275d2fd2ff1fc4414c not found: ID does not exist" Nov 02 19:30:05.237380 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.237353 5850 scope.go:122] "RemoveContainer" containerID="1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a" Nov 02 19:30:05.237605 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:30:05.237589 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a\": container with ID starting with 1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a not found: ID does not exist" containerID="1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a" Nov 02 19:30:05.237650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.237609 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a"} err="failed to get container status \"1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a\": rpc error: code = NotFound desc = could not find container \"1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a\": container with ID starting with 1e4d5eaaa6055084db760d07a0c0c07efaf969adf6675809867c0d98b1dadf0a not found: ID does not exist" Nov 02 19:30:05.237650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.237621 5850 scope.go:122] "RemoveContainer" containerID="069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9" Nov 02 19:30:05.237821 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:30:05.237806 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9\": container with ID starting with 069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9 not found: ID does not exist" containerID="069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9" Nov 02 19:30:05.237859 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.237824 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9"} err="failed to get container status \"069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9\": rpc error: code = NotFound desc = could not find container \"069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9\": container with ID starting with 069e26f07c6f7244b73589158459bf7e4a90d9f05a5139bcbac98d2e1b31d4b9 not found: ID does not exist" Nov 02 19:30:05.237859 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.237837 5850 scope.go:122] "RemoveContainer" containerID="e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af" Nov 02 19:30:05.238073 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:30:05.238054 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af\": container with ID starting with e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af not found: ID does not exist" containerID="e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af" Nov 02 19:30:05.238123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.238082 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af"} err="failed to get container status \"e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af\": rpc error: code = NotFound desc = could not find container \"e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af\": container with ID starting with e99d756579aecb1a7d9308e7f0de2b0691880663f59733c3a069c0e0a9c2c6af not found: ID does not exist" Nov 02 19:30:05.350372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.350330 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:30:05.401913 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.401884 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:30:05.402317 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402298 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:30:05.402609 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402596 5850 cpu_manager.go:414] "containerMap: removing container" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="alertmanager" Nov 02 19:30:05.402652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402610 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="alertmanager" Nov 02 19:30:05.402652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402625 5850 cpu_manager.go:414] "containerMap: removing container" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy-web" Nov 02 19:30:05.402652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402630 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy-web" Nov 02 19:30:05.402652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402644 5850 cpu_manager.go:414] "containerMap: removing container" podUID="2752b72d-0a38-4fdd-ab0f-905bae22cedf" containerName="console" Nov 02 19:30:05.402652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402649 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2752b72d-0a38-4fdd-ab0f-905bae22cedf" containerName="console" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402655 5850 cpu_manager.go:414] "containerMap: removing container" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="config-reloader" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402660 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="config-reloader" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402670 5850 cpu_manager.go:414] "containerMap: removing container" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="init-config-reloader" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402674 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="init-config-reloader" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402681 5850 cpu_manager.go:414] "containerMap: removing container" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="prom-label-proxy" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402686 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="prom-label-proxy" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402693 5850 cpu_manager.go:414] "containerMap: removing container" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402698 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402709 5850 cpu_manager.go:414] "containerMap: removing container" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy-metric" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402714 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy-metric" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402761 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402770 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy-metric" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402777 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="alertmanager" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402783 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="kube-rbac-proxy-web" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402789 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="2752b72d-0a38-4fdd-ab0f-905bae22cedf" containerName="console" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402794 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="config-reloader" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402800 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="init-config-reloader" Nov 02 19:30:05.402799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.402806 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" containerName="prom-label-proxy" Nov 02 19:30:05.444336 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.444260 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.499773 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.499742 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 19:30:05.499773 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.499742 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 19:30:05.499773 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.499742 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 19:30:05.500050 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.499894 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 19:30:05.532403 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532366 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfba8c33-8580-4b90-ae51-b31988a27692-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532403 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532407 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-web-config\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532428 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532513 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfba8c33-8580-4b90-ae51-b31988a27692-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532565 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-config-volume\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532591 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532609 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532630 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cfba8c33-8580-4b90-ae51-b31988a27692-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532893 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532748 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfba8c33-8580-4b90-ae51-b31988a27692-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532893 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532803 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.532893 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532824 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.533034 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532896 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfba8c33-8580-4b90-ae51-b31988a27692-config-out\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.533034 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.532944 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqd4\" (UniqueName: \"kubernetes.io/projected/cfba8c33-8580-4b90-ae51-b31988a27692-kube-api-access-cdqd4\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.550785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.550748 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 19:30:05.550971 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.550817 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-snhdz\"" Nov 02 19:30:05.550971 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.550820 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 19:30:05.550971 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.550856 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 19:30:05.551075 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.550992 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 19:30:05.551075 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.550993 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 19:30:05.552059 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.552041 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:30:05.633575 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633538 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.633575 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633579 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cfba8c33-8580-4b90-ae51-b31988a27692-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.633835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633601 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfba8c33-8580-4b90-ae51-b31988a27692-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.633835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633726 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.633835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633758 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.633835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633802 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfba8c33-8580-4b90-ae51-b31988a27692-config-out\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.634212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633848 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqd4\" (UniqueName: \"kubernetes.io/projected/cfba8c33-8580-4b90-ae51-b31988a27692-kube-api-access-cdqd4\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.634212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633890 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfba8c33-8580-4b90-ae51-b31988a27692-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.634212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633911 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-web-config\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.634212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633935 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.634212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.633961 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfba8c33-8580-4b90-ae51-b31988a27692-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.634212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.634004 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cfba8c33-8580-4b90-ae51-b31988a27692-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.634212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.634011 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-config-volume\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.634212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.634083 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.634973 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.634946 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfba8c33-8580-4b90-ae51-b31988a27692-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.635186 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.635160 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfba8c33-8580-4b90-ae51-b31988a27692-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.636991 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.636962 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.637082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.636987 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfba8c33-8580-4b90-ae51-b31988a27692-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.637082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.636987 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.637164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.637146 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.637309 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.637288 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.637766 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.637747 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfba8c33-8580-4b90-ae51-b31988a27692-config-out\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.638152 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.638134 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.638397 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.638379 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-web-config\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.638454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.638424 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cfba8c33-8580-4b90-ae51-b31988a27692-config-volume\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.692620 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.692582 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqd4\" (UniqueName: \"kubernetes.io/projected/cfba8c33-8580-4b90-ae51-b31988a27692-kube-api-access-cdqd4\") pod \"alertmanager-main-0\" (UID: \"cfba8c33-8580-4b90-ae51-b31988a27692\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:05.812468 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.812388 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4c68e9-066f-4de3-9657-27fa1831f327" path="/var/lib/kubelet/pods/aa4c68e9-066f-4de3-9657-27fa1831f327/volumes" Nov 02 19:30:05.844665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:05.844637 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:30:06.329869 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:06.329842 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:30:06.330118 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:30:06.330092 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfba8c33_8580_4b90_ae51_b31988a27692.slice/crio-2f98ab1f12b6d615e3721fa755e82f18bc476ecdda72ba0af91348570f321e9e WatchSource:0}: Error finding container 2f98ab1f12b6d615e3721fa755e82f18bc476ecdda72ba0af91348570f321e9e: Status 404 returned error can't find the container with id 2f98ab1f12b6d615e3721fa755e82f18bc476ecdda72ba0af91348570f321e9e Nov 02 19:30:07.188151 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:07.188117 5850 generic.go:356] "Generic (PLEG): container finished" podID="cfba8c33-8580-4b90-ae51-b31988a27692" containerID="554dd8dda5deb21f1bfe8827b26fd6ea9f2b50c50f1d14342f170493e968d8a1" exitCode=0 Nov 02 19:30:07.188561 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:07.188211 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfba8c33-8580-4b90-ae51-b31988a27692","Type":"ContainerDied","Data":"554dd8dda5deb21f1bfe8827b26fd6ea9f2b50c50f1d14342f170493e968d8a1"} Nov 02 19:30:07.188561 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:07.188281 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfba8c33-8580-4b90-ae51-b31988a27692","Type":"ContainerStarted","Data":"2f98ab1f12b6d615e3721fa755e82f18bc476ecdda72ba0af91348570f321e9e"} Nov 02 19:30:08.195671 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:08.195627 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfba8c33-8580-4b90-ae51-b31988a27692","Type":"ContainerStarted","Data":"52d904be210083c4f4dbdb2fa887e8ff7addb57146bf29573a8fdc54960abe2e"} Nov 02 19:30:08.196122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:08.195701 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfba8c33-8580-4b90-ae51-b31988a27692","Type":"ContainerStarted","Data":"14ad50b1db7931195fb9f306bc1534b4ee7482c77ce72a95bae46b31fb281787"} Nov 02 19:30:08.196122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:08.195736 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfba8c33-8580-4b90-ae51-b31988a27692","Type":"ContainerStarted","Data":"c6f19f67b88fa71c63df05ed297cfdc240b92ae7c7f93d367d9600509452bc24"} Nov 02 19:30:08.196122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:08.195762 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfba8c33-8580-4b90-ae51-b31988a27692","Type":"ContainerStarted","Data":"9728a8c87c55dedeffb44c12fe23cc8aa68258467e9d083ffee229f198304d61"} Nov 02 19:30:08.196122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:08.195785 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfba8c33-8580-4b90-ae51-b31988a27692","Type":"ContainerStarted","Data":"5a347659ddf513cdf0457c4e861b3775f1ed7e79c927ccacf78e8d782ce285ac"} Nov 02 19:30:08.196122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:08.195814 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cfba8c33-8580-4b90-ae51-b31988a27692","Type":"ContainerStarted","Data":"c000e62a8d3384b3f64fd4237acf49635befbdd7ed2807c2d8fb4531701b7e2f"} Nov 02 19:30:08.323218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:08.323159 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.309741351 podStartE2EDuration="3.323141403s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.400052ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:30:08.320371738 +0000 UTC m=+66.662760842" watchObservedRunningTime="2026-11-02 19:30:08.323141403 +0000 UTC m=+66.665530480" Nov 02 19:30:11.205206 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:11.205170 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:11.360633 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:11.360599 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:14.759694 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:30:14.759655 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:30:17.492736 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:17.492698 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:17.652014 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:17.651963 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:24.875336 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.875300 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-574f477cc-9pcl5"] Nov 02 19:30:24.877942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.877922 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:24.978598 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.978558 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574f477cc-9pcl5"] Nov 02 19:30:24.996790 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.996756 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-service-ca\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:24.996790 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.996797 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-console-config\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:24.996986 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.996814 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-trusted-ca-bundle\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:24.996986 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.996932 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-console-serving-cert\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:24.997053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.996999 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-oauth-serving-cert\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:24.997053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.997036 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccs45\" (UniqueName: \"kubernetes.io/projected/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-kube-api-access-ccs45\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:24.997110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.997056 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-tmp\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:24.997110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:24.997071 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-console-oauth-config\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.097789 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.097745 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-console-config\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.097789 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.097785 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-trusted-ca-bundle\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098039 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.097804 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-console-serving-cert\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098039 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.097829 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-oauth-serving-cert\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098039 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.097857 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccs45\" (UniqueName: \"kubernetes.io/projected/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-kube-api-access-ccs45\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098039 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.097875 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-tmp\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098039 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.097889 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-console-oauth-config\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098039 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.097932 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-service-ca\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098464 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.098435 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-tmp\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098744 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.098720 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-console-config\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.098723 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-oauth-serving-cert\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.098818 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-trusted-ca-bundle\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.098846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.098828 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-service-ca\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.100415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.100390 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-console-oauth-config\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.100559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.100543 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-console-serving-cert\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.133762 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.133695 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:25.157699 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.157667 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccs45\" (UniqueName: \"kubernetes.io/projected/53d48d20-2f13-4bd3-ad10-cd3c621a7f39-kube-api-access-ccs45\") pod \"console-574f477cc-9pcl5\" (UID: \"53d48d20-2f13-4bd3-ad10-cd3c621a7f39\") " pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.188527 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.188458 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:25.288545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.288508 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:25.505170 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.505086 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:30:25.555048 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:30:25.555004 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d48d20_2f13_4bd3_ad10_cd3c621a7f39.slice/crio-179360a4188da4c66cfadc3e3082265d90d215fc940bb887d7aaadef13d7dbc2 WatchSource:0}: Error finding container 179360a4188da4c66cfadc3e3082265d90d215fc940bb887d7aaadef13d7dbc2: Status 404 returned error can't find the container with id 179360a4188da4c66cfadc3e3082265d90d215fc940bb887d7aaadef13d7dbc2 Nov 02 19:30:25.555637 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:25.555574 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574f477cc-9pcl5"] Nov 02 19:30:26.270454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:26.270416 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574f477cc-9pcl5" event={"ID":"53d48d20-2f13-4bd3-ad10-cd3c621a7f39","Type":"ContainerStarted","Data":"d3e7a7c2e690c3864a09435569d1880f246f6f282395fb956c5112546a227375"} Nov 02 19:30:26.270862 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:26.270463 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574f477cc-9pcl5" event={"ID":"53d48d20-2f13-4bd3-ad10-cd3c621a7f39","Type":"ContainerStarted","Data":"179360a4188da4c66cfadc3e3082265d90d215fc940bb887d7aaadef13d7dbc2"} Nov 02 19:30:26.389212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:26.389159 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-574f477cc-9pcl5" podStartSLOduration=2.389141028 podStartE2EDuration="2.389141028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:30:26.388761012 +0000 UTC m=+84.731150092" watchObservedRunningTime="2026-11-02 19:30:26.389141028 +0000 UTC m=+84.731530108" Nov 02 19:30:35.188791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:35.188747 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:35.189185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:35.188866 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:35.193623 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:35.193593 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:35.305823 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:35.305794 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-574f477cc-9pcl5" Nov 02 19:30:35.459207 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:30:35.459124 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f584d8bb-wsxmb"] Nov 02 19:30:44.760514 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:30:44.760411 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:31:00.480251 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.480184 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-84f584d8bb-wsxmb" podUID="74918778-82ad-4bde-bdaf-7d913a496a04" containerName="console" containerID="cri-o://787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353" gracePeriod=15 Nov 02 19:31:00.721237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.721212 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:31:00.801370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.801274 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-oauth-config\") pod \"74918778-82ad-4bde-bdaf-7d913a496a04\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " Nov 02 19:31:00.801370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.801330 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74918778-82ad-4bde-bdaf-7d913a496a04-kube-api-access-l674d\" (UniqueName: \"kubernetes.io/projected/74918778-82ad-4bde-bdaf-7d913a496a04-kube-api-access-l674d\") pod \"74918778-82ad-4bde-bdaf-7d913a496a04\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " Nov 02 19:31:00.801370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.801352 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-console-config\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-console-config\") pod \"74918778-82ad-4bde-bdaf-7d913a496a04\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " Nov 02 19:31:00.801677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.801382 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74918778-82ad-4bde-bdaf-7d913a496a04-tmp\" (UniqueName: \"kubernetes.io/empty-dir/74918778-82ad-4bde-bdaf-7d913a496a04-tmp\") pod \"74918778-82ad-4bde-bdaf-7d913a496a04\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " Nov 02 19:31:00.801677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.801403 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-oauth-serving-cert\") pod \"74918778-82ad-4bde-bdaf-7d913a496a04\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " Nov 02 19:31:00.801677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.801420 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-service-ca\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-service-ca\") pod \"74918778-82ad-4bde-bdaf-7d913a496a04\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " Nov 02 19:31:00.801677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.801498 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-serving-cert\") pod \"74918778-82ad-4bde-bdaf-7d913a496a04\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " Nov 02 19:31:00.801677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.801645 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-trusted-ca-bundle\") pod \"74918778-82ad-4bde-bdaf-7d913a496a04\" (UID: \"74918778-82ad-4bde-bdaf-7d913a496a04\") " Nov 02 19:31:00.802195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.802112 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74918778-82ad-4bde-bdaf-7d913a496a04-tmp" pod "74918778-82ad-4bde-bdaf-7d913a496a04" (UID: "74918778-82ad-4bde-bdaf-7d913a496a04"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:31:00.802398 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.802353 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-oauth-serving-cert" pod "74918778-82ad-4bde-bdaf-7d913a496a04" (UID: "74918778-82ad-4bde-bdaf-7d913a496a04"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:31:00.802398 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.802385 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-console-config" pod "74918778-82ad-4bde-bdaf-7d913a496a04" (UID: "74918778-82ad-4bde-bdaf-7d913a496a04"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:31:00.802609 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.802442 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-service-ca" pod "74918778-82ad-4bde-bdaf-7d913a496a04" (UID: "74918778-82ad-4bde-bdaf-7d913a496a04"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:31:00.802609 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.802505 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-trusted-ca-bundle" pod "74918778-82ad-4bde-bdaf-7d913a496a04" (UID: "74918778-82ad-4bde-bdaf-7d913a496a04"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:31:00.803551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.803529 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-oauth-config" pod "74918778-82ad-4bde-bdaf-7d913a496a04" (UID: "74918778-82ad-4bde-bdaf-7d913a496a04"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:31:00.803990 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.803971 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74918778-82ad-4bde-bdaf-7d913a496a04-kube-api-access-l674d" pod "74918778-82ad-4bde-bdaf-7d913a496a04" (UID: "74918778-82ad-4bde-bdaf-7d913a496a04"). InnerVolumeSpecName "kube-api-access-l674d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:31:00.804268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.804241 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-serving-cert" pod "74918778-82ad-4bde-bdaf-7d913a496a04" (UID: "74918778-82ad-4bde-bdaf-7d913a496a04"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:31:00.902560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.902523 5850 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74918778-82ad-4bde-bdaf-7d913a496a04-tmp\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:31:00.902560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.902554 5850 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-oauth-serving-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:31:00.902560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.902564 5850 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:31:00.902805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.902575 5850 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-serving-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:31:00.902805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.902593 5850 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-trusted-ca-bundle\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:31:00.902805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.902602 5850 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74918778-82ad-4bde-bdaf-7d913a496a04-console-oauth-config\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:31:00.902805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.902610 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l674d\" (UniqueName: \"kubernetes.io/projected/74918778-82ad-4bde-bdaf-7d913a496a04-kube-api-access-l674d\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:31:00.902805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:00.902621 5850 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74918778-82ad-4bde-bdaf-7d913a496a04-console-config\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:31:01.398414 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.398379 5850 generic.go:356] "Generic (PLEG): container finished" podID="74918778-82ad-4bde-bdaf-7d913a496a04" containerID="787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353" exitCode=0 Nov 02 19:31:01.398614 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.398427 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f584d8bb-wsxmb" Nov 02 19:31:01.398614 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.398429 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f584d8bb-wsxmb" event={"ID":"74918778-82ad-4bde-bdaf-7d913a496a04","Type":"ContainerDied","Data":"787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353"} Nov 02 19:31:01.398614 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.398547 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f584d8bb-wsxmb" event={"ID":"74918778-82ad-4bde-bdaf-7d913a496a04","Type":"ContainerDied","Data":"8fcb6199b4db27b2b9a7b9ab6d75dbed53778f66a9e65f7613142024e58367d4"} Nov 02 19:31:01.398614 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.398573 5850 scope.go:122] "RemoveContainer" containerID="787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353" Nov 02 19:31:01.407284 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.407265 5850 scope.go:122] "RemoveContainer" containerID="787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353" Nov 02 19:31:01.407590 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:31:01.407570 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353\": container with ID starting with 787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353 not found: ID does not exist" containerID="787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353" Nov 02 19:31:01.407656 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.407597 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353"} err="failed to get container status \"787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353\": rpc error: code = NotFound desc = could not find container \"787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353\": container with ID starting with 787511dcf87c1f43f4e00859b744ff3d3a1ea565b1a29eeee34cd33898138353 not found: ID does not exist" Nov 02 19:31:01.565896 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.565861 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f584d8bb-wsxmb"] Nov 02 19:31:01.617426 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.617394 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84f584d8bb-wsxmb"] Nov 02 19:31:01.812431 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:01.812396 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74918778-82ad-4bde-bdaf-7d913a496a04" path="/var/lib/kubelet/pods/74918778-82ad-4bde-bdaf-7d913a496a04/volumes" Nov 02 19:31:04.245033 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:04.244999 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:04.547023 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:04.546930 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:14.761013 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:31:14.760971 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:31:44.759291 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:31:44.759249 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:31:49.548658 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:49.548618 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:49.818853 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:49.818767 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:50.687280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.687242 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22"] Nov 02 19:31:50.687722 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.687612 5850 cpu_manager.go:414] "containerMap: removing container" podUID="74918778-82ad-4bde-bdaf-7d913a496a04" containerName="console" Nov 02 19:31:50.687722 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.687628 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74918778-82ad-4bde-bdaf-7d913a496a04" containerName="console" Nov 02 19:31:50.687800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.687745 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="74918778-82ad-4bde-bdaf-7d913a496a04" containerName="console" Nov 02 19:31:50.690292 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.690271 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:50.744429 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.744389 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 19:31:50.744429 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.744390 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 19:31:50.796809 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.796780 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22"] Nov 02 19:31:50.826776 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.826736 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgr8h\" (UniqueName: \"kubernetes.io/projected/3c22d534-51e1-4641-8030-3c7f87b7566e-kube-api-access-zgr8h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:50.826950 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.826849 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:50.827000 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.826961 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:50.847786 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.847751 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-79lwf\"" Nov 02 19:31:50.921933 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.921893 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4"] Nov 02 19:31:50.924404 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.924386 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:50.928406 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.928373 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgr8h\" (UniqueName: \"kubernetes.io/projected/3c22d534-51e1-4641-8030-3c7f87b7566e-kube-api-access-zgr8h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:50.928588 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.928412 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:50.928588 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.928463 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:50.928861 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.928837 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:50.928899 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.928848 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:50.975705 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.975626 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:50.978837 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.978808 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 19:31:50.978837 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.978822 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 19:31:50.979011 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.978944 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 19:31:50.979011 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.978956 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 19:31:50.979098 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.979024 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 19:31:50.979098 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.979040 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 19:31:50.979171 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.979153 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gd6zm\"" Nov 02 19:31:50.987941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.987914 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4"] Nov 02 19:31:50.991089 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:50.991062 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgr8h\" (UniqueName: \"kubernetes.io/projected/3c22d534-51e1-4641-8030-3c7f87b7566e-kube-api-access-zgr8h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:51.003839 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.003807 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:31:51.029246 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.029204 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.029418 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.029261 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.029418 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.029377 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.029418 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.029410 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/309bda24-9929-45b4-a39e-fe847a47ec02-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.029611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.029428 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47h2\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-kube-api-access-h47h2\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.029611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.029527 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.029611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.029576 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.130286 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.130249 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.130286 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.130294 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/309bda24-9929-45b4-a39e-fe847a47ec02-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.130601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.130320 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47h2\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-kube-api-access-h47h2\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.130601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.130345 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.130601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.130390 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.130601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.130420 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.130601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.130457 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.131220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.131193 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.132846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.132819 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/309bda24-9929-45b4-a39e-fe847a47ec02-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.132955 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.132869 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.133020 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.132987 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.133020 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.132998 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.148819 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.148794 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:51.189595 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.189562 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.189881 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.189861 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47h2\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-kube-api-access-h47h2\") pod \"istiod-openshift-gateway-7876bfcb74-d2wm4\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.235109 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.235029 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:51.370703 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:31:51.370672 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c22d534_51e1_4641_8030_3c7f87b7566e.slice/crio-423b0310587f6451cbdd869698134676a37d6020060887532af23fba83ec1a81 WatchSource:0}: Error finding container 423b0310587f6451cbdd869698134676a37d6020060887532af23fba83ec1a81: Status 404 returned error can't find the container with id 423b0310587f6451cbdd869698134676a37d6020060887532af23fba83ec1a81 Nov 02 19:31:51.371779 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.371754 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22"] Nov 02 19:31:51.579749 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.579665 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" event={"ID":"3c22d534-51e1-4641-8030-3c7f87b7566e","Type":"ContainerStarted","Data":"423b0310587f6451cbdd869698134676a37d6020060887532af23fba83ec1a81"} Nov 02 19:31:51.605870 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:31:51.605837 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309bda24_9929_45b4_a39e_fe847a47ec02.slice/crio-53a16c6803e90ce3499e227621a6c5f190e4326d5f84443eb20f49259b1b3e3e WatchSource:0}: Error finding container 53a16c6803e90ce3499e227621a6c5f190e4326d5f84443eb20f49259b1b3e3e: Status 404 returned error can't find the container with id 53a16c6803e90ce3499e227621a6c5f190e4326d5f84443eb20f49259b1b3e3e Nov 02 19:31:51.605997 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.605892 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4"] Nov 02 19:31:51.906118 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:51.906073 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:52.585137 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:52.585095 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" event={"ID":"309bda24-9929-45b4-a39e-fe847a47ec02","Type":"ContainerStarted","Data":"53a16c6803e90ce3499e227621a6c5f190e4326d5f84443eb20f49259b1b3e3e"} Nov 02 19:31:56.502747 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:56.502701 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:31:56.503127 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:56.502778 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:31:56.604207 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:56.604166 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" event={"ID":"309bda24-9929-45b4-a39e-fe847a47ec02","Type":"ContainerStarted","Data":"a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad"} Nov 02 19:31:56.604403 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:56.604278 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:31:57.611136 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:57.611096 5850 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-d2wm4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:31:57.611634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:57.611166 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" podUID="309bda24-9929-45b4-a39e-fe847a47ec02" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:31:57.990154 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:57.990118 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:58.269358 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:58.269282 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:58.614314 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:58.614275 5850 generic.go:356] "Generic (PLEG): container finished" podID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerID="15e9f07d2370ec5eb060031c1f53ec339a7caaae899386729828a203d582827d" exitCode=0 Nov 02 19:31:58.614813 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:58.614360 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" event={"ID":"3c22d534-51e1-4641-8030-3c7f87b7566e","Type":"ContainerDied","Data":"15e9f07d2370ec5eb060031c1f53ec339a7caaae899386729828a203d582827d"} Nov 02 19:31:58.778225 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:58.778109 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" podStartSLOduration=3.884360889 podStartE2EDuration="8.778089256s" totalImagesPullingTime="4.893728367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:31:56.731778958 +0000 UTC m=+175.074168037" watchObservedRunningTime="2026-11-02 19:31:58.778089256 +0000 UTC m=+177.120478337" Nov 02 19:31:58.948619 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:58.948582 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:59.242120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:59.242078 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:31:59.404032 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:31:59.403996 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:00.610081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:00.610053 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:32:02.632196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:02.632156 5850 generic.go:356] "Generic (PLEG): container finished" podID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerID="e9b3f9ef7c204673661dbe0ebacf77a4313b8ff6044bf6c64b7044b5c89fba39" exitCode=0 Nov 02 19:32:02.632632 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:02.632241 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" event={"ID":"3c22d534-51e1-4641-8030-3c7f87b7566e","Type":"ContainerDied","Data":"e9b3f9ef7c204673661dbe0ebacf77a4313b8ff6044bf6c64b7044b5c89fba39"} Nov 02 19:32:03.004801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:03.004752 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:04.579842 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:04.579800 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:14.760566 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:32:14.760465 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:32:20.708211 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:20.708134 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" event={"ID":"3c22d534-51e1-4641-8030-3c7f87b7566e","Type":"ContainerStarted","Data":"5f5e01dedcdbd4a157e31b0585b27cca127e2e2084b6cd69eeb5a5d5e15dc398"} Nov 02 19:32:20.825459 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:20.825403 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" podStartSLOduration=2.044976436 podStartE2EDuration="30.825389036s" totalImagesPullingTime="28.7804126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:31:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:20.824731428 +0000 UTC m=+199.167120510" watchObservedRunningTime="2026-11-02 19:32:20.825389036 +0000 UTC m=+199.167778116" Nov 02 19:32:21.714068 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:21.714033 5850 generic.go:356] "Generic (PLEG): container finished" podID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerID="5f5e01dedcdbd4a157e31b0585b27cca127e2e2084b6cd69eeb5a5d5e15dc398" exitCode=0 Nov 02 19:32:21.714446 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:21.714111 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" event={"ID":"3c22d534-51e1-4641-8030-3c7f87b7566e","Type":"ContainerDied","Data":"5f5e01dedcdbd4a157e31b0585b27cca127e2e2084b6cd69eeb5a5d5e15dc398"} Nov 02 19:32:22.843564 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:22.843533 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:32:22.927015 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:22.926981 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c22d534-51e1-4641-8030-3c7f87b7566e-kube-api-access-zgr8h\" (UniqueName: \"kubernetes.io/projected/3c22d534-51e1-4641-8030-3c7f87b7566e-kube-api-access-zgr8h\") pod \"3c22d534-51e1-4641-8030-3c7f87b7566e\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " Nov 02 19:32:22.927015 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:22.927016 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-util\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-util\") pod \"3c22d534-51e1-4641-8030-3c7f87b7566e\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " Nov 02 19:32:22.927219 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:22.927108 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-bundle\") pod \"3c22d534-51e1-4641-8030-3c7f87b7566e\" (UID: \"3c22d534-51e1-4641-8030-3c7f87b7566e\") " Nov 02 19:32:22.927527 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:22.927501 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-bundle" pod "3c22d534-51e1-4641-8030-3c7f87b7566e" (UID: "3c22d534-51e1-4641-8030-3c7f87b7566e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:22.929229 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:22.929198 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c22d534-51e1-4641-8030-3c7f87b7566e-kube-api-access-zgr8h" pod "3c22d534-51e1-4641-8030-3c7f87b7566e" (UID: "3c22d534-51e1-4641-8030-3c7f87b7566e"). InnerVolumeSpecName "kube-api-access-zgr8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:22.931340 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:22.931316 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-util" pod "3c22d534-51e1-4641-8030-3c7f87b7566e" (UID: "3c22d534-51e1-4641-8030-3c7f87b7566e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:23.028796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:23.028708 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgr8h\" (UniqueName: \"kubernetes.io/projected/3c22d534-51e1-4641-8030-3c7f87b7566e-kube-api-access-zgr8h\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.028796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:23.028737 5850 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-util\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.028796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:23.028747 5850 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c22d534-51e1-4641-8030-3c7f87b7566e-bundle\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.722765 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:23.722731 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" event={"ID":"3c22d534-51e1-4641-8030-3c7f87b7566e","Type":"ContainerDied","Data":"423b0310587f6451cbdd869698134676a37d6020060887532af23fba83ec1a81"} Nov 02 19:32:23.722934 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:23.722803 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="423b0310587f6451cbdd869698134676a37d6020060887532af23fba83ec1a81" Nov 02 19:32:23.722934 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:23.722743 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jgn22" Nov 02 19:32:25.755874 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:25.755837 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:26.043221 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.043140 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:26.423599 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.423560 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:26.647719 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.647683 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl"] Nov 02 19:32:26.648018 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.648006 5850 cpu_manager.go:414] "containerMap: removing container" podUID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerName="extract" Nov 02 19:32:26.648066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.648019 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerName="extract" Nov 02 19:32:26.648066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.648040 5850 cpu_manager.go:414] "containerMap: removing container" podUID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerName="util" Nov 02 19:32:26.648066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.648046 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerName="util" Nov 02 19:32:26.648066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.648052 5850 cpu_manager.go:414] "containerMap: removing container" podUID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerName="pull" Nov 02 19:32:26.648066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.648057 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerName="pull" Nov 02 19:32:26.648231 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.648107 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c22d534-51e1-4641-8030-3c7f87b7566e" containerName="extract" Nov 02 19:32:26.650788 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.650748 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.703342 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.703232 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:26.762849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.762810 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-webhook-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.762849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.762859 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-tmp\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.763290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.762887 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fwb\" (UniqueName: \"kubernetes.io/projected/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-kube-api-access-x6fwb\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.763290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.762992 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.763290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.763028 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-manifests\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.763290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.763064 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-metrics-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.771511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.771458 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 19:32:26.771694 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.771578 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 19:32:26.772447 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.772432 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 19:32:26.772447 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.772444 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 19:32:26.772577 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.772496 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 19:32:26.772671 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.772658 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lqjs8\"" Nov 02 19:32:26.863832 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.863796 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.864029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.863841 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-manifests\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.864029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.863881 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-metrics-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.864029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.863936 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-webhook-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.864029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.863971 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-tmp\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.864029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.864012 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fwb\" (UniqueName: \"kubernetes.io/projected/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-kube-api-access-x6fwb\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.864329 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.864308 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-manifests\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.866173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.866147 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-tmp\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.866508 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.866492 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-webhook-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.866553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.866533 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.867316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.867297 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-metrics-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.924779 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.924739 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fwb\" (UniqueName: \"kubernetes.io/projected/72d3d2f3-63e2-4d80-b8a0-36db012aeb52-kube-api-access-x6fwb\") pod \"opendatahub-operator-controller-manager-5985f94fb7-7zwrl\" (UID: \"72d3d2f3-63e2-4d80-b8a0-36db012aeb52\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.965057 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.964974 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:26.965372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.965346 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl"] Nov 02 19:32:26.973870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:26.973837 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:27.231337 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:27.231254 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:27.457000 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:32:27.456972 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d3d2f3_63e2_4d80_b8a0_36db012aeb52.slice/crio-1098b3cb6429a48207d2fddd28d4e94709d11f5853ce421939acbeba2e8d2ad1 WatchSource:0}: Error finding container 1098b3cb6429a48207d2fddd28d4e94709d11f5853ce421939acbeba2e8d2ad1: Status 404 returned error can't find the container with id 1098b3cb6429a48207d2fddd28d4e94709d11f5853ce421939acbeba2e8d2ad1 Nov 02 19:32:27.459395 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:27.459374 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl"] Nov 02 19:32:27.737541 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:27.737504 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" event={"ID":"72d3d2f3-63e2-4d80-b8a0-36db012aeb52","Type":"ContainerStarted","Data":"1098b3cb6429a48207d2fddd28d4e94709d11f5853ce421939acbeba2e8d2ad1"} Nov 02 19:32:27.805907 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:27.805875 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:27.979612 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:27.979576 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:30.390231 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:30.390196 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:30.555966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:30.555934 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:30.953656 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:30.953563 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:31.757968 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:31.757935 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" event={"ID":"72d3d2f3-63e2-4d80-b8a0-36db012aeb52","Type":"ContainerStarted","Data":"99ccf1713eda71ca15d6045ea29483d893289a5b591a1305dd674ebcabb6fe0d"} Nov 02 19:32:32.765057 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:32.765018 5850 generic.go:356] "Generic (PLEG): container finished" podID="72d3d2f3-63e2-4d80-b8a0-36db012aeb52" containerID="99ccf1713eda71ca15d6045ea29483d893289a5b591a1305dd674ebcabb6fe0d" exitCode=0 Nov 02 19:32:32.765452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:32.765108 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" event={"ID":"72d3d2f3-63e2-4d80-b8a0-36db012aeb52","Type":"ContainerDied","Data":"99ccf1713eda71ca15d6045ea29483d893289a5b591a1305dd674ebcabb6fe0d"} Nov 02 19:32:33.771603 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:33.771570 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" event={"ID":"72d3d2f3-63e2-4d80-b8a0-36db012aeb52","Type":"ContainerStarted","Data":"19fbdf3c3b88248f822a92aaa2c1e24d85f4d58060cfd5c7113c35ef1eddd6ae"} Nov 02 19:32:33.771994 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:33.771706 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:34.012305 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:34.012220 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:34.054768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:34.054707 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" podStartSLOduration=2.868125982 podStartE2EDuration="8.054689597s" totalImagesPullingTime="4.763876273s" totalInitContainerRuntime="422.687342ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:34.002452062 +0000 UTC m=+212.344841142" watchObservedRunningTime="2026-11-02 19:32:34.054689597 +0000 UTC m=+212.397078676" Nov 02 19:32:36.086699 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:36.086666 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:36.570179 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:36.570142 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:37.358905 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:37.358873 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:43.188209 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:43.188170 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:44.350302 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:44.350272 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:44.760580 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:32:44.760538 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:32:44.777461 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:44.777431 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-7zwrl" Nov 02 19:32:45.048047 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:45.047964 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:45.237097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:45.237062 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:47.415661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:47.415627 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:47.946280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:47.946249 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:50.174383 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:50.174350 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:50.334846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:50.334808 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:50.560059 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:50.559983 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:52.997195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:52.997162 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:54.367398 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:54.367345 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:54.527009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:54.526970 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:54.748008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:54.747894 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:57.067207 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:57.067174 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:32:57.274737 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:32:57.274704 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:00.182542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:00.182507 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:00.386007 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:00.385974 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:00.645238 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:00.645199 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:00.844951 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:00.844918 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:01.366797 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:01.366769 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:01.585307 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:01.585272 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:01.792777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:01.792687 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:01.975194 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:01.975168 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:02.197834 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:02.197798 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:02.440851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:02.440814 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:02.639141 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:02.639099 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:03.014282 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.014206 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:03.214081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.214044 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:03.301313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.301239 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg"] Nov 02 19:33:03.306608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.306574 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" Nov 02 19:33:03.368123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.368090 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg"] Nov 02 19:33:03.397018 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.396984 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pxsn\" (UniqueName: \"kubernetes.io/projected/35e51273-6a03-4bc6-9a9e-6bd031f4c1d1-kube-api-access-5pxsn\") pod \"cert-manager-cainjector-5c57bbd4fc-jwmpg\" (UID: \"35e51273-6a03-4bc6-9a9e-6bd031f4c1d1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" Nov 02 19:33:03.397190 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.397047 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35e51273-6a03-4bc6-9a9e-6bd031f4c1d1-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jwmpg\" (UID: \"35e51273-6a03-4bc6-9a9e-6bd031f4c1d1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" Nov 02 19:33:03.418727 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.418696 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 19:33:03.418727 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.418715 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 19:33:03.428041 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.428013 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-ww858\"" Nov 02 19:33:03.459888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.459851 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:03.498413 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.498376 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxsn\" (UniqueName: \"kubernetes.io/projected/35e51273-6a03-4bc6-9a9e-6bd031f4c1d1-kube-api-access-5pxsn\") pod \"cert-manager-cainjector-5c57bbd4fc-jwmpg\" (UID: \"35e51273-6a03-4bc6-9a9e-6bd031f4c1d1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" Nov 02 19:33:03.498605 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.498432 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35e51273-6a03-4bc6-9a9e-6bd031f4c1d1-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jwmpg\" (UID: \"35e51273-6a03-4bc6-9a9e-6bd031f4c1d1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" Nov 02 19:33:03.557211 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.557138 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35e51273-6a03-4bc6-9a9e-6bd031f4c1d1-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jwmpg\" (UID: \"35e51273-6a03-4bc6-9a9e-6bd031f4c1d1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" Nov 02 19:33:03.557420 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.557397 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxsn\" (UniqueName: \"kubernetes.io/projected/35e51273-6a03-4bc6-9a9e-6bd031f4c1d1-kube-api-access-5pxsn\") pod \"cert-manager-cainjector-5c57bbd4fc-jwmpg\" (UID: \"35e51273-6a03-4bc6-9a9e-6bd031f4c1d1\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" Nov 02 19:33:03.633879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:03.633845 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" Nov 02 19:33:04.032752 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.032724 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg"] Nov 02 19:33:04.033583 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:33:04.033552 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35e51273_6a03_4bc6_9a9e_6bd031f4c1d1.slice/crio-efc246349db326a3b6005b1f4ee3ca8214e839d7f0e7a314b27e7b9f23bb7db4 WatchSource:0}: Error finding container efc246349db326a3b6005b1f4ee3ca8214e839d7f0e7a314b27e7b9f23bb7db4: Status 404 returned error can't find the container with id efc246349db326a3b6005b1f4ee3ca8214e839d7f0e7a314b27e7b9f23bb7db4 Nov 02 19:33:04.203469 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.203426 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:04.239610 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.239576 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hm8t2"] Nov 02 19:33:04.281798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.281720 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:04.302557 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.302527 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hm8t2"] Nov 02 19:33:04.306821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.306796 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqhz\" (UniqueName: \"kubernetes.io/projected/82d948e5-336e-4363-a0c3-ab310cab8a84-kube-api-access-mpqhz\") pod \"limitador-limitador-74dfcfb894-hm8t2\" (UID: \"82d948e5-336e-4363-a0c3-ab310cab8a84\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:04.306940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.306853 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82d948e5-336e-4363-a0c3-ab310cab8a84-config-file\") pod \"limitador-limitador-74dfcfb894-hm8t2\" (UID: \"82d948e5-336e-4363-a0c3-ab310cab8a84\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:04.336310 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.336279 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f9djg\"" Nov 02 19:33:04.336796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.336780 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 19:33:04.336894 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.336874 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 19:33:04.336944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.336884 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 19:33:04.407663 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.407630 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82d948e5-336e-4363-a0c3-ab310cab8a84-config-file\") pod \"limitador-limitador-74dfcfb894-hm8t2\" (UID: \"82d948e5-336e-4363-a0c3-ab310cab8a84\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:04.407857 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.407715 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqhz\" (UniqueName: \"kubernetes.io/projected/82d948e5-336e-4363-a0c3-ab310cab8a84-kube-api-access-mpqhz\") pod \"limitador-limitador-74dfcfb894-hm8t2\" (UID: \"82d948e5-336e-4363-a0c3-ab310cab8a84\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:04.408521 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.408489 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82d948e5-336e-4363-a0c3-ab310cab8a84-config-file\") pod \"limitador-limitador-74dfcfb894-hm8t2\" (UID: \"82d948e5-336e-4363-a0c3-ab310cab8a84\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:04.446310 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.446273 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:04.477667 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.477626 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hm8t2"] Nov 02 19:33:04.478406 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.478386 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqhz\" (UniqueName: \"kubernetes.io/projected/82d948e5-336e-4363-a0c3-ab310cab8a84-kube-api-access-mpqhz\") pod \"limitador-limitador-74dfcfb894-hm8t2\" (UID: \"82d948e5-336e-4363-a0c3-ab310cab8a84\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:04.613606 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.613516 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:04.673284 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.673253 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65657466b-r9pv9"] Nov 02 19:33:04.673962 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.673867 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:04.676950 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.676927 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.710852 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.710816 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f99a447-9c59-4e48-866c-1ace82f01793-tmp\") pod \"kube-auth-proxy-65657466b-r9pv9\" (UID: \"4f99a447-9c59-4e48-866c-1ace82f01793\") " pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.711040 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.710870 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f99a447-9c59-4e48-866c-1ace82f01793-tls-certs\") pod \"kube-auth-proxy-65657466b-r9pv9\" (UID: \"4f99a447-9c59-4e48-866c-1ace82f01793\") " pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.711040 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.710986 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xxq\" (UniqueName: \"kubernetes.io/projected/4f99a447-9c59-4e48-866c-1ace82f01793-kube-api-access-q7xxq\") pod \"kube-auth-proxy-65657466b-r9pv9\" (UID: \"4f99a447-9c59-4e48-866c-1ace82f01793\") " pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.739088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.739048 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65657466b-r9pv9"] Nov 02 19:33:04.742419 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.742383 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 19:33:04.742419 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.742383 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7db5j\"" Nov 02 19:33:04.742665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.742499 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 19:33:04.811562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.811526 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xxq\" (UniqueName: \"kubernetes.io/projected/4f99a447-9c59-4e48-866c-1ace82f01793-kube-api-access-q7xxq\") pod \"kube-auth-proxy-65657466b-r9pv9\" (UID: \"4f99a447-9c59-4e48-866c-1ace82f01793\") " pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.811723 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.811584 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f99a447-9c59-4e48-866c-1ace82f01793-tmp\") pod \"kube-auth-proxy-65657466b-r9pv9\" (UID: \"4f99a447-9c59-4e48-866c-1ace82f01793\") " pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.811723 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.811611 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f99a447-9c59-4e48-866c-1ace82f01793-tls-certs\") pod \"kube-auth-proxy-65657466b-r9pv9\" (UID: \"4f99a447-9c59-4e48-866c-1ace82f01793\") " pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.813982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.813950 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f99a447-9c59-4e48-866c-1ace82f01793-tmp\") pod \"kube-auth-proxy-65657466b-r9pv9\" (UID: \"4f99a447-9c59-4e48-866c-1ace82f01793\") " pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.814179 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.814161 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f99a447-9c59-4e48-866c-1ace82f01793-tls-certs\") pod \"kube-auth-proxy-65657466b-r9pv9\" (UID: \"4f99a447-9c59-4e48-866c-1ace82f01793\") " pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.872693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.872658 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xxq\" (UniqueName: \"kubernetes.io/projected/4f99a447-9c59-4e48-866c-1ace82f01793-kube-api-access-q7xxq\") pod \"kube-auth-proxy-65657466b-r9pv9\" (UID: \"4f99a447-9c59-4e48-866c-1ace82f01793\") " pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:04.884061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.884032 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:04.902323 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.902288 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" event={"ID":"35e51273-6a03-4bc6-9a9e-6bd031f4c1d1","Type":"ContainerStarted","Data":"efc246349db326a3b6005b1f4ee3ca8214e839d7f0e7a314b27e7b9f23bb7db4"} Nov 02 19:33:04.985194 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:33:04.985154 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d948e5_336e_4363_a0c3_ab310cab8a84.slice/crio-40428ca0328eb7d11d8453a46fe9361aa59416a7a4d927c70afe186b8a9e38d0 WatchSource:0}: Error finding container 40428ca0328eb7d11d8453a46fe9361aa59416a7a4d927c70afe186b8a9e38d0: Status 404 returned error can't find the container with id 40428ca0328eb7d11d8453a46fe9361aa59416a7a4d927c70afe186b8a9e38d0 Nov 02 19:33:04.986536 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.986514 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hm8t2"] Nov 02 19:33:04.990236 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:04.990212 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" Nov 02 19:33:05.077339 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:05.077302 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:05.252376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:05.252299 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:05.364822 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:33:05.364781 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f99a447_9c59_4e48_866c_1ace82f01793.slice/crio-8710de354a89db6d971961eec0f0d82ca3ed21faff9783088197ff1eae8a64ea WatchSource:0}: Error finding container 8710de354a89db6d971961eec0f0d82ca3ed21faff9783088197ff1eae8a64ea: Status 404 returned error can't find the container with id 8710de354a89db6d971961eec0f0d82ca3ed21faff9783088197ff1eae8a64ea Nov 02 19:33:05.365107 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:05.365088 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65657466b-r9pv9"] Nov 02 19:33:05.417539 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:05.417501 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:05.600632 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:05.600524 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:05.773933 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:05.773879 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:05.909430 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:05.909387 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" event={"ID":"4f99a447-9c59-4e48-866c-1ace82f01793","Type":"ContainerStarted","Data":"8710de354a89db6d971961eec0f0d82ca3ed21faff9783088197ff1eae8a64ea"} Nov 02 19:33:05.910731 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:05.910701 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" event={"ID":"82d948e5-336e-4363-a0c3-ab310cab8a84","Type":"ContainerStarted","Data":"40428ca0328eb7d11d8453a46fe9361aa59416a7a4d927c70afe186b8a9e38d0"} Nov 02 19:33:05.945868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:05.945830 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:09.933856 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:09.933815 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" event={"ID":"82d948e5-336e-4363-a0c3-ab310cab8a84","Type":"ContainerStarted","Data":"b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14"} Nov 02 19:33:09.934649 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:09.934598 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:09.936700 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:09.936165 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" event={"ID":"4f99a447-9c59-4e48-866c-1ace82f01793","Type":"ContainerStarted","Data":"5f130263b5baf06325f54a34a0f342e9bac279eb1abe0c0587d6fa59e5efd8a7"} Nov 02 19:33:10.052677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:10.052623 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" podStartSLOduration=1.5496117360000001 podStartE2EDuration="6.05260976s" totalImagesPullingTime="4.502998024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:10.052410729 +0000 UTC m=+248.394799809" watchObservedRunningTime="2026-11-02 19:33:10.05260976 +0000 UTC m=+248.394998839" Nov 02 19:33:10.170287 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:10.170230 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65657466b-r9pv9" podStartSLOduration=1.8282847869999999 podStartE2EDuration="6.170215679s" totalImagesPullingTime="4.341930892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:10.169709883 +0000 UTC m=+248.512098963" watchObservedRunningTime="2026-11-02 19:33:10.170215679 +0000 UTC m=+248.512604750" Nov 02 19:33:10.586844 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:10.586808 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:10.752453 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:10.752402 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:10.942750 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:10.942705 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" event={"ID":"35e51273-6a03-4bc6-9a9e-6bd031f4c1d1","Type":"ContainerStarted","Data":"80eeb86c2f67d33923b7e7fc9a55977bb63d959a88176ba2dce0a9b0ae16da36"} Nov 02 19:33:11.080866 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:11.080813 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jwmpg" podStartSLOduration=2.110126514 podStartE2EDuration="8.080799378s" totalImagesPullingTime="5.970672864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:11.080252374 +0000 UTC m=+249.422641449" watchObservedRunningTime="2026-11-02 19:33:11.080799378 +0000 UTC m=+249.423188457" Nov 02 19:33:11.133296 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:11.133260 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:11.616053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:11.616017 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:11.783979 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:11.783937 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:14.760440 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:33:14.760396 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:33:15.799233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:15.799193 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:15.970063 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:15.970013 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:17.227222 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:17.227185 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:17.393253 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:17.393221 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:18.391227 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:18.391191 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:18.558838 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:18.558806 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:20.175320 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:20.175283 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:21.728949 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:21.728917 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:21.903011 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:21.902971 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:21.947982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:21.947953 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:22.836673 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:22.836638 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hm8t2"] Nov 02 19:33:22.837188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:22.836852 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" podUID="82d948e5-336e-4363-a0c3-ab310cab8a84" containerName="limitador" containerID="cri-o://b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14" gracePeriod=30 Nov 02 19:33:23.376086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.376059 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:23.493061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.492979 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d948e5-336e-4363-a0c3-ab310cab8a84-kube-api-access-mpqhz\" (UniqueName: \"kubernetes.io/projected/82d948e5-336e-4363-a0c3-ab310cab8a84-kube-api-access-mpqhz\") pod \"82d948e5-336e-4363-a0c3-ab310cab8a84\" (UID: \"82d948e5-336e-4363-a0c3-ab310cab8a84\") " Nov 02 19:33:23.493061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.493035 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d948e5-336e-4363-a0c3-ab310cab8a84-config-file\" (UniqueName: \"kubernetes.io/configmap/82d948e5-336e-4363-a0c3-ab310cab8a84-config-file\") pod \"82d948e5-336e-4363-a0c3-ab310cab8a84\" (UID: \"82d948e5-336e-4363-a0c3-ab310cab8a84\") " Nov 02 19:33:23.493456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.493429 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d948e5-336e-4363-a0c3-ab310cab8a84-config-file" pod "82d948e5-336e-4363-a0c3-ab310cab8a84" (UID: "82d948e5-336e-4363-a0c3-ab310cab8a84"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:33:23.495625 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.495593 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d948e5-336e-4363-a0c3-ab310cab8a84-kube-api-access-mpqhz" pod "82d948e5-336e-4363-a0c3-ab310cab8a84" (UID: "82d948e5-336e-4363-a0c3-ab310cab8a84"). InnerVolumeSpecName "kube-api-access-mpqhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:33:23.594235 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.594193 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpqhz\" (UniqueName: \"kubernetes.io/projected/82d948e5-336e-4363-a0c3-ab310cab8a84-kube-api-access-mpqhz\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:33:23.594235 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.594229 5850 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82d948e5-336e-4363-a0c3-ab310cab8a84-config-file\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:33:23.997442 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.997408 5850 generic.go:356] "Generic (PLEG): container finished" podID="82d948e5-336e-4363-a0c3-ab310cab8a84" containerID="b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14" exitCode=0 Nov 02 19:33:23.997952 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.997454 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" Nov 02 19:33:23.997952 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.997461 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" event={"ID":"82d948e5-336e-4363-a0c3-ab310cab8a84","Type":"ContainerDied","Data":"b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14"} Nov 02 19:33:23.997952 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.997542 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hm8t2" event={"ID":"82d948e5-336e-4363-a0c3-ab310cab8a84","Type":"ContainerDied","Data":"40428ca0328eb7d11d8453a46fe9361aa59416a7a4d927c70afe186b8a9e38d0"} Nov 02 19:33:23.997952 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:23.997578 5850 scope.go:122] "RemoveContainer" containerID="b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14" Nov 02 19:33:24.006285 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:24.006264 5850 scope.go:122] "RemoveContainer" containerID="b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14" Nov 02 19:33:24.006633 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:33:24.006611 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14\": container with ID starting with b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14 not found: ID does not exist" containerID="b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14" Nov 02 19:33:24.006715 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:24.006643 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14"} err="failed to get container status \"b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14\": rpc error: code = NotFound desc = could not find container \"b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14\": container with ID starting with b42c1e39dec8c702a11dae89e450cbc864bc9a73953fbe29cbc61f51450cee14 not found: ID does not exist" Nov 02 19:33:24.165379 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:24.165342 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hm8t2"] Nov 02 19:33:24.216992 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:24.216957 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hm8t2"] Nov 02 19:33:25.813192 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:25.813163 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d948e5-336e-4363-a0c3-ab310cab8a84" path="/var/lib/kubelet/pods/82d948e5-336e-4363-a0c3-ab310cab8a84/volumes" Nov 02 19:33:28.200544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:28.200506 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:28.359687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:28.359649 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:34.754388 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:34.754336 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:34.907756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:34.907711 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:35.390238 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:35.390194 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:35.554821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:35.554770 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:35.981938 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:35.981903 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:36.083070 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.083034 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt"] Nov 02 19:33:36.083411 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.083396 5850 cpu_manager.go:414] "containerMap: removing container" podUID="82d948e5-336e-4363-a0c3-ab310cab8a84" containerName="limitador" Nov 02 19:33:36.083453 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.083414 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d948e5-336e-4363-a0c3-ab310cab8a84" containerName="limitador" Nov 02 19:33:36.083523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.083511 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d948e5-336e-4363-a0c3-ab310cab8a84" containerName="limitador" Nov 02 19:33:36.086955 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.086938 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:36.103902 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.103872 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt8d8\" (UniqueName: \"kubernetes.io/projected/2c0f7860-82ee-4546-b523-7bc62e32d1e4-kube-api-access-zt8d8\") pod \"postgres-7cdbf6dc44-dx6gt\" (UID: \"2c0f7860-82ee-4546-b523-7bc62e32d1e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:36.104056 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.103947 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2c0f7860-82ee-4546-b523-7bc62e32d1e4-data\") pod \"postgres-7cdbf6dc44-dx6gt\" (UID: \"2c0f7860-82ee-4546-b523-7bc62e32d1e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:36.142795 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.142764 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt"] Nov 02 19:33:36.152682 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.152639 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:36.193456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.193420 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:33:36.193456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.193456 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:33:36.193691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.193500 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 19:33:36.193691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.193534 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-67g54\"" Nov 02 19:33:36.204809 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.204771 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2c0f7860-82ee-4546-b523-7bc62e32d1e4-data\") pod \"postgres-7cdbf6dc44-dx6gt\" (UID: \"2c0f7860-82ee-4546-b523-7bc62e32d1e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:36.204974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.204866 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt8d8\" (UniqueName: \"kubernetes.io/projected/2c0f7860-82ee-4546-b523-7bc62e32d1e4-kube-api-access-zt8d8\") pod \"postgres-7cdbf6dc44-dx6gt\" (UID: \"2c0f7860-82ee-4546-b523-7bc62e32d1e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:36.205201 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.205180 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2c0f7860-82ee-4546-b523-7bc62e32d1e4-data\") pod \"postgres-7cdbf6dc44-dx6gt\" (UID: \"2c0f7860-82ee-4546-b523-7bc62e32d1e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:36.263518 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.263417 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt8d8\" (UniqueName: \"kubernetes.io/projected/2c0f7860-82ee-4546-b523-7bc62e32d1e4-kube-api-access-zt8d8\") pod \"postgres-7cdbf6dc44-dx6gt\" (UID: \"2c0f7860-82ee-4546-b523-7bc62e32d1e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:36.310725 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.310686 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:36.399578 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.399535 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:36.763623 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:33:36.763589 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0f7860_82ee_4546_b523_7bc62e32d1e4.slice/crio-7b5a532035f57ad9bc305cae6ffc70c2031680bbb255b2c58721d2b46601841f WatchSource:0}: Error finding container 7b5a532035f57ad9bc305cae6ffc70c2031680bbb255b2c58721d2b46601841f: Status 404 returned error can't find the container with id 7b5a532035f57ad9bc305cae6ffc70c2031680bbb255b2c58721d2b46601841f Nov 02 19:33:36.763948 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:36.763925 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt"] Nov 02 19:33:37.052735 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:37.052645 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" event={"ID":"2c0f7860-82ee-4546-b523-7bc62e32d1e4","Type":"ContainerStarted","Data":"7b5a532035f57ad9bc305cae6ffc70c2031680bbb255b2c58721d2b46601841f"} Nov 02 19:33:37.344462 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:37.344380 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:37.512184 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:37.512120 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:41.167220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:41.167016 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:41.332176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:41.332144 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:41.491176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:41.491094 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:42.360977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:42.360932 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:42.513188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:42.513148 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:44.780560 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:33:44.780518 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:33:46.095445 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:46.095411 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" event={"ID":"2c0f7860-82ee-4546-b523-7bc62e32d1e4","Type":"ContainerStarted","Data":"6fe14316a92e3d0b4ea20004573496890d1b48bb0fd09c3639df375962b34520"} Nov 02 19:33:46.095870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:46.095513 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:46.212548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:46.212465 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" podStartSLOduration=1.807962528 podStartE2EDuration="10.212451248s" totalImagesPullingTime="8.40448872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:46.211267343 +0000 UTC m=+284.553656423" watchObservedRunningTime="2026-11-02 19:33:46.212451248 +0000 UTC m=+284.554840327" Nov 02 19:33:49.195405 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:49.195348 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:49.351011 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:49.350975 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:51.471227 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:51.471195 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:51.622873 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:51.622841 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:52.128573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:52.128542 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dx6gt" Nov 02 19:33:56.920663 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:56.920606 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:57.073021 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:57.072984 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:57.764810 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:57.764772 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:57.925035 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:57.924990 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:59.105817 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:59.105782 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:59.265038 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:59.264981 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:59.419101 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:59.419069 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:33:59.577856 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:33:59.577800 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:03.805195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:03.805164 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:04.244646 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:04.244616 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:04.421847 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:04.421811 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:05.212793 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:05.212757 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:05.536388 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:05.536298 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:07.196439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:07.196386 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:07.583846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:07.583757 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:14.765504 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:34:14.765444 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:34:16.760739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:16.760710 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86"] Nov 02 19:34:16.903055 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:16.903004 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86"] Nov 02 19:34:16.903233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:16.903108 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.084371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.084274 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aed97a99-5750-40b1-be06-80341d05defe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.084371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.084322 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aed97a99-5750-40b1-be06-80341d05defe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.084634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.084436 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed97a99-5750-40b1-be06-80341d05defe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.084634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.084493 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aed97a99-5750-40b1-be06-80341d05defe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.084634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.084617 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aed97a99-5750-40b1-be06-80341d05defe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.084767 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.084652 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed97a99-5750-40b1-be06-80341d05defe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.084767 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.084688 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5d26\" (UniqueName: \"kubernetes.io/projected/aed97a99-5750-40b1-be06-80341d05defe-kube-api-access-r5d26\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.185905 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.185862 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed97a99-5750-40b1-be06-80341d05defe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.185905 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.185910 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aed97a99-5750-40b1-be06-80341d05defe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.186180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.185965 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aed97a99-5750-40b1-be06-80341d05defe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.186180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.185990 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed97a99-5750-40b1-be06-80341d05defe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.186180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.186016 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5d26\" (UniqueName: \"kubernetes.io/projected/aed97a99-5750-40b1-be06-80341d05defe-kube-api-access-r5d26\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.186180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.186083 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aed97a99-5750-40b1-be06-80341d05defe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.186180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.186119 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aed97a99-5750-40b1-be06-80341d05defe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.186868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.186835 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aed97a99-5750-40b1-be06-80341d05defe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.188505 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.188466 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aed97a99-5750-40b1-be06-80341d05defe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.188618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.188571 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aed97a99-5750-40b1-be06-80341d05defe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.188684 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.188638 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed97a99-5750-40b1-be06-80341d05defe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.188684 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.188668 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aed97a99-5750-40b1-be06-80341d05defe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.245192 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.245160 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed97a99-5750-40b1-be06-80341d05defe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.245426 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.245404 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5d26\" (UniqueName: \"kubernetes.io/projected/aed97a99-5750-40b1-be06-80341d05defe-kube-api-access-r5d26\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rsp86\" (UID: \"aed97a99-5750-40b1-be06-80341d05defe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.532950 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.532913 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:17.979715 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:34:17.979685 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed97a99_5750_40b1_be06_80341d05defe.slice/crio-154af6d31f57e92b7ca7ec51b4386c0a223dd3959acd813d5cef027108dc9856 WatchSource:0}: Error finding container 154af6d31f57e92b7ca7ec51b4386c0a223dd3959acd813d5cef027108dc9856: Status 404 returned error can't find the container with id 154af6d31f57e92b7ca7ec51b4386c0a223dd3959acd813d5cef027108dc9856 Nov 02 19:34:17.980784 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.980759 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86"] Nov 02 19:34:17.981759 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.981726 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:34:17.981864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:17.981799 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:34:18.230522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:18.230405 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" event={"ID":"aed97a99-5750-40b1-be06-80341d05defe","Type":"ContainerStarted","Data":"1bda166355ffff886565b02e7848cb138a13635cd5755f63208ca176aad83c89"} Nov 02 19:34:18.230522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:18.230484 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" event={"ID":"aed97a99-5750-40b1-be06-80341d05defe","Type":"ContainerStarted","Data":"154af6d31f57e92b7ca7ec51b4386c0a223dd3959acd813d5cef027108dc9856"} Nov 02 19:34:18.230705 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:18.230605 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:18.407801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:18.407740 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" podStartSLOduration=2.407722635 podStartE2EDuration="2.407722635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:18.355702616 +0000 UTC m=+316.698091708" watchObservedRunningTime="2026-11-02 19:34:18.407722635 +0000 UTC m=+316.750111714" Nov 02 19:34:19.238137 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:19.238102 5850 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rsp86 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:34:19.238819 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:19.238160 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" podUID="aed97a99-5750-40b1-be06-80341d05defe" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:34:21.653675 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:21.653643 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:21.830519 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:21.830461 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:21.998230 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:21.998139 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:22.157392 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.157358 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:22.237200 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.237168 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rsp86" Nov 02 19:34:22.315131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.315049 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:22.457233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.457203 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4"] Nov 02 19:34:22.457431 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.457409 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" podUID="309bda24-9929-45b4-a39e-fe847a47ec02" containerName="discovery" containerID="cri-o://a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad" gracePeriod=30 Nov 02 19:34:22.471369 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.471336 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:22.636377 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.636341 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:22.707521 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.707461 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:34:22.730803 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.730760 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-cacerts\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-cacerts\") pod \"309bda24-9929-45b4-a39e-fe847a47ec02\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " Nov 02 19:34:22.731005 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.730843 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-kube-api-access-h47h2\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-kube-api-access-h47h2\") pod \"309bda24-9929-45b4-a39e-fe847a47ec02\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " Nov 02 19:34:22.731005 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.730937 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-ca-configmap\") pod \"309bda24-9929-45b4-a39e-fe847a47ec02\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " Nov 02 19:34:22.731005 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.730986 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-istio-token\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-istio-token\") pod \"309bda24-9929-45b4-a39e-fe847a47ec02\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " Nov 02 19:34:22.731178 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.731021 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-dns-cert\") pod \"309bda24-9929-45b4-a39e-fe847a47ec02\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " Nov 02 19:34:22.731178 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.731073 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-kubeconfig\") pod \"309bda24-9929-45b4-a39e-fe847a47ec02\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " Nov 02 19:34:22.731178 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.731113 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309bda24-9929-45b4-a39e-fe847a47ec02-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/309bda24-9929-45b4-a39e-fe847a47ec02-local-certs\") pod \"309bda24-9929-45b4-a39e-fe847a47ec02\" (UID: \"309bda24-9929-45b4-a39e-fe847a47ec02\") " Nov 02 19:34:22.731340 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.731315 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-ca-configmap" pod "309bda24-9929-45b4-a39e-fe847a47ec02" (UID: "309bda24-9929-45b4-a39e-fe847a47ec02"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:34:22.731449 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.731414 5850 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-ca-configmap\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:22.733302 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.733269 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-cacerts" pod "309bda24-9929-45b4-a39e-fe847a47ec02" (UID: "309bda24-9929-45b4-a39e-fe847a47ec02"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:34:22.733439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.733367 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-kubeconfig" pod "309bda24-9929-45b4-a39e-fe847a47ec02" (UID: "309bda24-9929-45b4-a39e-fe847a47ec02"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:34:22.733640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.733514 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-kube-api-access-h47h2" pod "309bda24-9929-45b4-a39e-fe847a47ec02" (UID: "309bda24-9929-45b4-a39e-fe847a47ec02"). InnerVolumeSpecName "kube-api-access-h47h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:34:22.733640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.733535 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-istio-token" pod "309bda24-9929-45b4-a39e-fe847a47ec02" (UID: "309bda24-9929-45b4-a39e-fe847a47ec02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:34:22.733802 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.733777 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309bda24-9929-45b4-a39e-fe847a47ec02-local-certs" pod "309bda24-9929-45b4-a39e-fe847a47ec02" (UID: "309bda24-9929-45b4-a39e-fe847a47ec02"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:34:22.733954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.733929 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-dns-cert" pod "309bda24-9929-45b4-a39e-fe847a47ec02" (UID: "309bda24-9929-45b4-a39e-fe847a47ec02"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:34:22.831876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.831795 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:22.831876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.831830 5850 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-csr-dns-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:22.831876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.831845 5850 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-istio-kubeconfig\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:22.831876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.831859 5850 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/309bda24-9929-45b4-a39e-fe847a47ec02-local-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:22.831876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.831873 5850 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/309bda24-9929-45b4-a39e-fe847a47ec02-cacerts\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:22.832181 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:22.831886 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h47h2\" (UniqueName: \"kubernetes.io/projected/309bda24-9929-45b4-a39e-fe847a47ec02-kube-api-access-h47h2\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:23.254004 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.253972 5850 generic.go:356] "Generic (PLEG): container finished" podID="309bda24-9929-45b4-a39e-fe847a47ec02" containerID="a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad" exitCode=0 Nov 02 19:34:23.254173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.254016 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" event={"ID":"309bda24-9929-45b4-a39e-fe847a47ec02","Type":"ContainerDied","Data":"a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad"} Nov 02 19:34:23.254173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.254022 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" Nov 02 19:34:23.254173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.254059 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4" event={"ID":"309bda24-9929-45b4-a39e-fe847a47ec02","Type":"ContainerDied","Data":"53a16c6803e90ce3499e227621a6c5f190e4326d5f84443eb20f49259b1b3e3e"} Nov 02 19:34:23.254173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.254079 5850 scope.go:122] "RemoveContainer" containerID="a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad" Nov 02 19:34:23.263229 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.263210 5850 scope.go:122] "RemoveContainer" containerID="a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad" Nov 02 19:34:23.263567 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:34:23.263545 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad\": container with ID starting with a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad not found: ID does not exist" containerID="a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad" Nov 02 19:34:23.263639 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.263579 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad"} err="failed to get container status \"a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad\": rpc error: code = NotFound desc = could not find container \"a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad\": container with ID starting with a373419d0ca1239623982e1efaa4b5fddf81ea36a742dc4288337675381925ad not found: ID does not exist" Nov 02 19:34:23.423125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.423034 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4"] Nov 02 19:34:23.427360 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.427330 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d2wm4"] Nov 02 19:34:23.713014 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.712938 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:23.813433 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:23.813397 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309bda24-9929-45b4-a39e-fe847a47ec02" path="/var/lib/kubelet/pods/309bda24-9929-45b4-a39e-fe847a47ec02/volumes" Nov 02 19:34:28.468469 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:28.468435 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:28.629434 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:28.629398 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:28.827983 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:28.827893 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:28.983026 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:28.982988 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:29.184559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:29.184527 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:29.345013 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:29.344979 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:29.501018 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:29.500928 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:29.656689 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:29.656657 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:37.379930 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:37.379890 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:37.555996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:37.555965 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:39.447459 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:39.447412 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:39.603116 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:39.603083 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:39.759681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:39.759601 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:39.919497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:39.919441 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:40.534487 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.534444 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx"] Nov 02 19:34:40.534861 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.534807 5850 cpu_manager.go:414] "containerMap: removing container" podUID="309bda24-9929-45b4-a39e-fe847a47ec02" containerName="discovery" Nov 02 19:34:40.534861 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.534818 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309bda24-9929-45b4-a39e-fe847a47ec02" containerName="discovery" Nov 02 19:34:40.534947 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.534897 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="309bda24-9929-45b4-a39e-fe847a47ec02" containerName="discovery" Nov 02 19:34:40.537339 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.537324 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.593596 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.593560 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 19:34:40.593849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.593562 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 19:34:40.593982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.593573 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sg6rf\"" Nov 02 19:34:40.595189 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.595165 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx"] Nov 02 19:34:40.596919 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.596892 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-metrics-certs\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.597035 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.596945 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f38399-6a7d-42ba-9609-59b55d1be41a-openshift-service-ca\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.597097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.597076 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-maas-api-tls\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.597197 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.597175 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkb9\" (UniqueName: \"kubernetes.io/projected/51f38399-6a7d-42ba-9609-59b55d1be41a-kube-api-access-qkkb9\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.697919 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.697876 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-metrics-certs\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.698100 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.697932 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f38399-6a7d-42ba-9609-59b55d1be41a-openshift-service-ca\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.698100 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.697971 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-maas-api-tls\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.698100 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.697989 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkb9\" (UniqueName: \"kubernetes.io/projected/51f38399-6a7d-42ba-9609-59b55d1be41a-kube-api-access-qkkb9\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.698674 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.698651 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f38399-6a7d-42ba-9609-59b55d1be41a-openshift-service-ca\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.700354 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.700333 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-metrics-certs\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.700540 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.700516 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-maas-api-tls\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.756610 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.756572 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkb9\" (UniqueName: \"kubernetes.io/projected/51f38399-6a7d-42ba-9609-59b55d1be41a-kube-api-access-qkkb9\") pod \"maas-api-6d669c7fcd-kpflx\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:40.849167 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:40.849073 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:41.216557 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:34:41.216527 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f38399_6a7d_42ba_9609_59b55d1be41a.slice/crio-5632f79e4034e67f2a08e9376a57224b796098bb4e6fcd3e1ad75646566f9bd3 WatchSource:0}: Error finding container 5632f79e4034e67f2a08e9376a57224b796098bb4e6fcd3e1ad75646566f9bd3: Status 404 returned error can't find the container with id 5632f79e4034e67f2a08e9376a57224b796098bb4e6fcd3e1ad75646566f9bd3 Nov 02 19:34:41.217376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:41.217352 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx"] Nov 02 19:34:41.217861 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:41.217847 5850 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:34:41.334981 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:41.334945 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" event={"ID":"51f38399-6a7d-42ba-9609-59b55d1be41a","Type":"ContainerStarted","Data":"5632f79e4034e67f2a08e9376a57224b796098bb4e6fcd3e1ad75646566f9bd3"} Nov 02 19:34:44.356218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:44.356174 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" event={"ID":"51f38399-6a7d-42ba-9609-59b55d1be41a","Type":"ContainerStarted","Data":"16ffd108110fe8e7f7bb7e095b4f58e471c67bde39d6ccc8e7aa70b4a43351a6"} Nov 02 19:34:44.356626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:44.356282 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:44.474081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:44.473974 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" podStartSLOduration=1.559662814 podStartE2EDuration="4.473957399s" totalImagesPullingTime="2.914294585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:44.473551042 +0000 UTC m=+342.815940123" watchObservedRunningTime="2026-11-02 19:34:44.473957399 +0000 UTC m=+342.816346517" Nov 02 19:34:44.625233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:44.625202 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:44.762046 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:34:44.761966 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:34:44.779988 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:44.779957 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:44.937189 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:44.937144 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:46.515016 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.514981 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k"] Nov 02 19:34:46.517593 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.517572 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.571882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.571845 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:34:46.571882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.571860 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 19:34:46.572069 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.571965 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 19:34:46.623628 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.623598 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 19:34:46.623842 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.623668 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 19:34:46.623842 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.623721 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-cd22z\"" Nov 02 19:34:46.624984 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.624961 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k"] Nov 02 19:34:46.654000 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.653968 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-cert\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.654131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.654008 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jmwz\" (UniqueName: \"kubernetes.io/projected/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-kube-api-access-9jmwz\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.654131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.654112 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-manager-config\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.654205 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.654155 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-metrics-certs\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.755427 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.755384 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-manager-config\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.755622 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.755433 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-metrics-certs\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.755622 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.755496 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-cert\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.755622 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.755519 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jmwz\" (UniqueName: \"kubernetes.io/projected/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-kube-api-access-9jmwz\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.756254 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.756228 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-manager-config\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.757983 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.757948 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-cert\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.758104 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.758082 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-metrics-certs\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.817417 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.817337 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jmwz\" (UniqueName: \"kubernetes.io/projected/e69d37a1-04bf-4f5b-833e-ba5e0d29ec07-kube-api-access-9jmwz\") pod \"jobset-controller-manager-6484845675-5qc8k\" (UID: \"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07\") " pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:46.829219 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:46.829181 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:47.196969 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:34:47.196926 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69d37a1_04bf_4f5b_833e_ba5e0d29ec07.slice/crio-71393ca4591e584e2558c2cb712228072c0953c66da42c0739c1ba4d2a2b6972 WatchSource:0}: Error finding container 71393ca4591e584e2558c2cb712228072c0953c66da42c0739c1ba4d2a2b6972: Status 404 returned error can't find the container with id 71393ca4591e584e2558c2cb712228072c0953c66da42c0739c1ba4d2a2b6972 Nov 02 19:34:47.197459 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:47.197438 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k"] Nov 02 19:34:47.370893 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:47.370855 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" event={"ID":"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07","Type":"ContainerStarted","Data":"71393ca4591e584e2558c2cb712228072c0953c66da42c0739c1ba4d2a2b6972"} Nov 02 19:34:50.366371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:50.366342 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:34:50.900833 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:50.900795 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:51.135893 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:51.135852 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:34:52.396700 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:52.396656 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" event={"ID":"e69d37a1-04bf-4f5b-833e-ba5e0d29ec07","Type":"ContainerStarted","Data":"e9a5f02265ee46a2e03cb55702fb44892e3d19fa7b1e8e5352275655193d22e5"} Nov 02 19:34:52.397233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:52.396750 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:34:52.512978 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:34:52.512927 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" podStartSLOduration=1.689068595 podStartE2EDuration="6.512914976s" totalImagesPullingTime="4.823846381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:52.51253755 +0000 UTC m=+350.854926627" watchObservedRunningTime="2026-11-02 19:34:52.512914976 +0000 UTC m=+350.855304056" Nov 02 19:35:01.238767 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.238730 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx"] Nov 02 19:35:01.239159 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.239007 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" podUID="51f38399-6a7d-42ba-9609-59b55d1be41a" containerName="maas-api" containerID="cri-o://16ffd108110fe8e7f7bb7e095b4f58e471c67bde39d6ccc8e7aa70b4a43351a6" gracePeriod=30 Nov 02 19:35:01.438132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.438100 5850 generic.go:356] "Generic (PLEG): container finished" podID="51f38399-6a7d-42ba-9609-59b55d1be41a" containerID="16ffd108110fe8e7f7bb7e095b4f58e471c67bde39d6ccc8e7aa70b4a43351a6" exitCode=0 Nov 02 19:35:01.438308 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.438151 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" event={"ID":"51f38399-6a7d-42ba-9609-59b55d1be41a","Type":"ContainerDied","Data":"16ffd108110fe8e7f7bb7e095b4f58e471c67bde39d6ccc8e7aa70b4a43351a6"} Nov 02 19:35:01.484102 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.484075 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:35:01.600300 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.600215 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f38399-6a7d-42ba-9609-59b55d1be41a-kube-api-access-qkkb9\" (UniqueName: \"kubernetes.io/projected/51f38399-6a7d-42ba-9609-59b55d1be41a-kube-api-access-qkkb9\") pod \"51f38399-6a7d-42ba-9609-59b55d1be41a\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " Nov 02 19:35:01.600496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.600322 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f38399-6a7d-42ba-9609-59b55d1be41a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f38399-6a7d-42ba-9609-59b55d1be41a-openshift-service-ca\") pod \"51f38399-6a7d-42ba-9609-59b55d1be41a\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " Nov 02 19:35:01.600496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.600342 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-maas-api-tls\") pod \"51f38399-6a7d-42ba-9609-59b55d1be41a\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " Nov 02 19:35:01.600496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.600360 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-metrics-certs\") pod \"51f38399-6a7d-42ba-9609-59b55d1be41a\" (UID: \"51f38399-6a7d-42ba-9609-59b55d1be41a\") " Nov 02 19:35:01.600780 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.600746 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f38399-6a7d-42ba-9609-59b55d1be41a-openshift-service-ca" pod "51f38399-6a7d-42ba-9609-59b55d1be41a" (UID: "51f38399-6a7d-42ba-9609-59b55d1be41a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:35:01.603467 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.603256 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f38399-6a7d-42ba-9609-59b55d1be41a-kube-api-access-qkkb9" pod "51f38399-6a7d-42ba-9609-59b55d1be41a" (UID: "51f38399-6a7d-42ba-9609-59b55d1be41a"). InnerVolumeSpecName "kube-api-access-qkkb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:35:01.603467 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.603337 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-maas-api-tls" pod "51f38399-6a7d-42ba-9609-59b55d1be41a" (UID: "51f38399-6a7d-42ba-9609-59b55d1be41a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:35:01.603467 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.603408 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-metrics-certs" pod "51f38399-6a7d-42ba-9609-59b55d1be41a" (UID: "51f38399-6a7d-42ba-9609-59b55d1be41a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:35:01.701298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.701253 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkkb9\" (UniqueName: \"kubernetes.io/projected/51f38399-6a7d-42ba-9609-59b55d1be41a-kube-api-access-qkkb9\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:35:01.701298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.701290 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f38399-6a7d-42ba-9609-59b55d1be41a-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:35:01.701298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.701302 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:35:01.701564 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:01.701312 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f38399-6a7d-42ba-9609-59b55d1be41a-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:35:02.443145 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:02.443105 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" event={"ID":"51f38399-6a7d-42ba-9609-59b55d1be41a","Type":"ContainerDied","Data":"5632f79e4034e67f2a08e9376a57224b796098bb4e6fcd3e1ad75646566f9bd3"} Nov 02 19:35:02.443145 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:02.443131 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx" Nov 02 19:35:02.443733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:02.443193 5850 scope.go:122] "RemoveContainer" containerID="16ffd108110fe8e7f7bb7e095b4f58e471c67bde39d6ccc8e7aa70b4a43351a6" Nov 02 19:35:02.612748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:02.612708 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx"] Nov 02 19:35:02.616207 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:02.616178 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kpflx"] Nov 02 19:35:03.406809 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:03.406774 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6484845675-5qc8k" Nov 02 19:35:03.813429 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:03.813334 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f38399-6a7d-42ba-9609-59b55d1be41a" path="/var/lib/kubelet/pods/51f38399-6a7d-42ba-9609-59b55d1be41a/volumes" Nov 02 19:35:05.544035 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:05.544000 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:35:14.761304 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:35:14.761227 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:35:15.517856 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:15.517809 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:35:15.688242 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:15.688213 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:35:16.837256 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:16.837225 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:35:17.027119 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:35:17.027085 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:35:44.760555 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:35:44.760513 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:36:05.869691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:05.869655 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:09.751107 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:09.751070 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:09.921837 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:09.921803 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:14.761028 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:36:14.760993 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:36:30.729737 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:30.729684 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:34.348117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:34.348083 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:37.186128 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.186094 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5"] Nov 02 19:36:37.186530 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.186456 5850 cpu_manager.go:414] "containerMap: removing container" podUID="51f38399-6a7d-42ba-9609-59b55d1be41a" containerName="maas-api" Nov 02 19:36:37.186530 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.186467 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f38399-6a7d-42ba-9609-59b55d1be41a" containerName="maas-api" Nov 02 19:36:37.186618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.186541 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f38399-6a7d-42ba-9609-59b55d1be41a" containerName="maas-api" Nov 02 19:36:37.188753 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.188736 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.242997 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.242925 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 19:36:37.242997 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.242992 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 19:36:37.292303 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.292272 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:37.294330 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.294310 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 19:36:37.294408 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.294318 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xwhqg\"" Nov 02 19:36:37.295421 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.295400 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5"] Nov 02 19:36:37.304863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.304829 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.305067 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.304886 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.305067 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.304999 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bf093b-465e-4039-a1a9-fa51b4262cd7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.305067 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.305051 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.305224 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.305098 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.305291 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.305268 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9tm9\" (UniqueName: \"kubernetes.io/projected/10bf093b-465e-4039-a1a9-fa51b4262cd7-kube-api-access-k9tm9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.406618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.406576 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.406618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.406628 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.406884 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.406812 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bf093b-465e-4039-a1a9-fa51b4262cd7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.406884 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.406840 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.406884 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.406875 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.407047 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.407021 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.407047 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.407031 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9tm9\" (UniqueName: \"kubernetes.io/projected/10bf093b-465e-4039-a1a9-fa51b4262cd7-kube-api-access-k9tm9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.407144 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.407082 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.407180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.407159 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.409139 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.409113 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bf093b-465e-4039-a1a9-fa51b4262cd7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.409346 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.409332 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bf093b-465e-4039-a1a9-fa51b4262cd7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.466595 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.466515 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9tm9\" (UniqueName: \"kubernetes.io/projected/10bf093b-465e-4039-a1a9-fa51b4262cd7-kube-api-access-k9tm9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ks6x5\" (UID: \"10bf093b-465e-4039-a1a9-fa51b4262cd7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.500313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.500271 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:37.872986 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:36:37.872954 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bf093b_465e_4039_a1a9_fa51b4262cd7.slice/crio-4dc35a065b36f88afd8eca168b2ed5c1ccf93c2b81138f3905822d4e7893d24b WatchSource:0}: Error finding container 4dc35a065b36f88afd8eca168b2ed5c1ccf93c2b81138f3905822d4e7893d24b: Status 404 returned error can't find the container with id 4dc35a065b36f88afd8eca168b2ed5c1ccf93c2b81138f3905822d4e7893d24b Nov 02 19:36:37.874324 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:37.874301 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5"] Nov 02 19:36:38.866023 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:38.864110 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" event={"ID":"10bf093b-465e-4039-a1a9-fa51b4262cd7","Type":"ContainerStarted","Data":"4dc35a065b36f88afd8eca168b2ed5c1ccf93c2b81138f3905822d4e7893d24b"} Nov 02 19:36:39.869424 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:39.869381 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" event={"ID":"10bf093b-465e-4039-a1a9-fa51b4262cd7","Type":"ContainerStarted","Data":"2a26558905af64a6c98093f4f198518136841d466ae4b74899dac1c859fefc6a"} Nov 02 19:36:39.986501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:39.986419 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" podStartSLOduration=1.431018215 podStartE2EDuration="2.986406493s" totalImagesPullingTime="1.555388278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:36:39.986053359 +0000 UTC m=+458.328442438" watchObservedRunningTime="2026-11-02 19:36:39.986406493 +0000 UTC m=+458.328795572" Nov 02 19:36:40.501360 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:40.501325 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:41.172446 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:41.172402 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:44.614489 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.614446 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf"] Nov 02 19:36:44.618025 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.618001 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.725449 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.725407 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 19:36:44.726639 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.726610 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf"] Nov 02 19:36:44.747992 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.747956 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:44.762708 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:36:44.762665 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:36:44.777955 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.777920 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.778122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.777963 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.778122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.778037 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8694\" (UniqueName: \"kubernetes.io/projected/6b8c1541-a097-42a4-8094-ce28470d7582-kube-api-access-l8694\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.778194 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.778116 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.778194 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.778180 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.778262 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.778201 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8c1541-a097-42a4-8094-ce28470d7582-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.879410 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.879365 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.879638 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.879426 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8694\" (UniqueName: \"kubernetes.io/projected/6b8c1541-a097-42a4-8094-ce28470d7582-kube-api-access-l8694\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.879638 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.879498 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.879638 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.879556 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.879638 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.879583 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8c1541-a097-42a4-8094-ce28470d7582-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.879855 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.879662 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.879855 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.879827 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.879921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.879872 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.880235 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.880216 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.882001 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.881981 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8c1541-a097-42a4-8094-ce28470d7582-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.882194 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.882174 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8c1541-a097-42a4-8094-ce28470d7582-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:44.939872 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:44.939832 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8694\" (UniqueName: \"kubernetes.io/projected/6b8c1541-a097-42a4-8094-ce28470d7582-kube-api-access-l8694\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf\" (UID: \"6b8c1541-a097-42a4-8094-ce28470d7582\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:45.231168 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:45.231058 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:45.598411 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:36:45.598376 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8c1541_a097_42a4_8094_ce28470d7582.slice/crio-35c511620ff94036779c8d9b9c4b4b56d5fe21c8a41d31a9af95f1ee60af0027 WatchSource:0}: Error finding container 35c511620ff94036779c8d9b9c4b4b56d5fe21c8a41d31a9af95f1ee60af0027: Status 404 returned error can't find the container with id 35c511620ff94036779c8d9b9c4b4b56d5fe21c8a41d31a9af95f1ee60af0027 Nov 02 19:36:45.600094 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:45.600068 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf"] Nov 02 19:36:45.897608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:45.897576 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" event={"ID":"6b8c1541-a097-42a4-8094-ce28470d7582","Type":"ContainerStarted","Data":"35c511620ff94036779c8d9b9c4b4b56d5fe21c8a41d31a9af95f1ee60af0027"} Nov 02 19:36:46.903809 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:46.903766 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" event={"ID":"6b8c1541-a097-42a4-8094-ce28470d7582","Type":"ContainerStarted","Data":"9e8382df6cf9b8acff0f8e63a0bf74c50d667e3efa517db162d26c2e7ed547d8"} Nov 02 19:36:47.022909 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:47.022858 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" podStartSLOduration=2.413076936 podStartE2EDuration="3.022843139s" totalImagesPullingTime="609.766203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:36:47.021548332 +0000 UTC m=+465.363937408" watchObservedRunningTime="2026-11-02 19:36:47.022843139 +0000 UTC m=+465.365232219" Nov 02 19:36:47.232243 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:47.232152 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:47.501106 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:47.501012 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:47.513679 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:47.513649 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:47.919344 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:47.919313 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ks6x5" Nov 02 19:36:48.494416 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:48.494382 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:52.219049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:52.219012 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:55.231653 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:55.231594 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:55.244119 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:55.244086 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:55.955422 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:55.955382 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf" Nov 02 19:36:56.764527 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.764489 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq"] Nov 02 19:36:56.768381 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.768359 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.825608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.825578 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq"] Nov 02 19:36:56.875139 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.875095 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 19:36:56.891551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.891519 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.891662 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.891572 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.891704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.891668 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.891739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.891702 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.891739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.891731 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcmrk\" (UniqueName: \"kubernetes.io/projected/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-kube-api-access-xcmrk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.891847 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.891765 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.893601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.893575 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:36:56.992895 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.992855 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.993097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.992927 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.993097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.992960 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.993097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.992981 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.993097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.993022 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcmrk\" (UniqueName: \"kubernetes.io/projected/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-kube-api-access-xcmrk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.993097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.993050 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.993370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.993350 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.993435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.993410 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.993435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.993422 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.995411 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.995395 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:56.995579 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:56.995561 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:57.051444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:57.051367 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcmrk\" (UniqueName: \"kubernetes.io/projected/db3b64b1-d8e5-4e64-a5c4-97158401ed5c-kube-api-access-xcmrk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq\" (UID: \"db3b64b1-d8e5-4e64-a5c4-97158401ed5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:57.079800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:57.079758 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:57.449364 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:36:57.449327 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb3b64b1_d8e5_4e64_a5c4_97158401ed5c.slice/crio-5e6862e86982858305960d035cbf0963b42aa1af5a0256e7f793946f7b0f849e WatchSource:0}: Error finding container 5e6862e86982858305960d035cbf0963b42aa1af5a0256e7f793946f7b0f849e: Status 404 returned error can't find the container with id 5e6862e86982858305960d035cbf0963b42aa1af5a0256e7f793946f7b0f849e Nov 02 19:36:57.450222 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:57.450199 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq"] Nov 02 19:36:57.955843 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:57.955810 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" event={"ID":"db3b64b1-d8e5-4e64-a5c4-97158401ed5c","Type":"ContainerStarted","Data":"5e6862e86982858305960d035cbf0963b42aa1af5a0256e7f793946f7b0f849e"} Nov 02 19:36:58.962201 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:58.962158 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" event={"ID":"db3b64b1-d8e5-4e64-a5c4-97158401ed5c","Type":"ContainerStarted","Data":"1405b7bc9ad4452108708e8734baabaad757a1305076e40d0059bb6c9bc2b3c0"} Nov 02 19:36:59.079887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:59.079846 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:36:59.080573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:36:59.080534 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podStartSLOduration=2.434796426 podStartE2EDuration="3.080519669s" totalImagesPullingTime="645.723243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:36:59.079804332 +0000 UTC m=+477.422193412" watchObservedRunningTime="2026-11-02 19:36:59.080519669 +0000 UTC m=+477.422908748" Nov 02 19:37:01.101146 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.101114 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr"] Nov 02 19:37:01.105527 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.105492 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.128949 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.128913 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.128949 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.128952 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad964b2-ea0b-4893-abc2-1376231e0e50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.129140 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.129029 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.129140 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.129054 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cc79\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-kube-api-access-8cc79\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.129217 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.129129 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.129217 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.129168 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.129217 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.129189 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.129321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.129218 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.129360 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.129320 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.159828 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.159796 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qmwl4\"" Nov 02 19:37:01.213032 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.212997 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr"] Nov 02 19:37:01.229777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.229738 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cc79\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-kube-api-access-8cc79\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.229777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.229779 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230025 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.229804 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230025 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.229823 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230025 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.229850 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230025 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.229911 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230025 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.229930 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.230076 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad964b2-ea0b-4893-abc2-1376231e0e50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.230157 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.230248 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.230259 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230431 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.230327 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230431 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.230401 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.230843 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.230823 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad964b2-ea0b-4893-abc2-1376231e0e50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.232308 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.232282 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.232580 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.232555 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.288844 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.288813 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.289115 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.289091 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cc79\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-kube-api-access-8cc79\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xwqbr\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.419724 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.419679 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:01.861679 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:37:01.861624 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad964b2_ea0b_4893_abc2_1376231e0e50.slice/crio-6db8cda59922715e88a1420e806bb74abad55e61b5733387cc43cb4771220e9e WatchSource:0}: Error finding container 6db8cda59922715e88a1420e806bb74abad55e61b5733387cc43cb4771220e9e: Status 404 returned error can't find the container with id 6db8cda59922715e88a1420e806bb74abad55e61b5733387cc43cb4771220e9e Nov 02 19:37:01.863759 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.863730 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr"] Nov 02 19:37:01.976259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:01.976223 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" event={"ID":"8ad964b2-ea0b-4893-abc2-1376231e0e50","Type":"ContainerStarted","Data":"6db8cda59922715e88a1420e806bb74abad55e61b5733387cc43cb4771220e9e"} Nov 02 19:37:05.887273 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:05.887235 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:37:05.887640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:05.887308 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:37:05.887640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:05.887340 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:37:05.997914 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:05.997868 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" event={"ID":"8ad964b2-ea0b-4893-abc2-1376231e0e50","Type":"ContainerStarted","Data":"dbc0746e2401f934a8a4521a1236f1a8b1d8ddfca71f32697ea81e55214e49f5"} Nov 02 19:37:06.195412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:06.195319 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:37:06.424537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:06.424180 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 19:37:06.424537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:06.424246 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 19:37:06.424537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:06.424521 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:06.519554 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:06.517968 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:37:07.080245 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:07.080209 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:37:07.093062 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:07.093031 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:37:07.214300 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:07.214235 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podStartSLOduration=2.19163117 podStartE2EDuration="6.214214413s" totalImagesPullingTime="4.022583243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:37:06.221859695 +0000 UTC m=+484.564248775" watchObservedRunningTime="2026-11-02 19:37:07.214214413 +0000 UTC m=+485.556603495" Nov 02 19:37:07.427499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:07.420976 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 19:37:07.427499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:07.421042 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 19:37:08.043023 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:08.042994 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:37:08.421960 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:08.421918 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 19:37:08.422979 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:08.422561 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 19:37:09.424991 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:09.424949 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 19:37:09.425639 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:09.425606 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 19:37:10.426499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:10.425967 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 19:37:10.426499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:10.426079 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 19:37:11.427496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:11.426908 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 19:37:11.427496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:11.426991 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 19:37:12.420232 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:12.420194 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 19:37:12.420415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:12.420250 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 19:37:13.421048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:13.421012 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 19:37:13.421450 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:13.421077 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 19:37:14.424755 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:14.424724 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:14.425203 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:14.424989 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:14.425617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:14.425600 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:37:14.763114 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:37:14.763020 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:37:16.225237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.225201 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6"] Nov 02 19:37:16.229276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.229253 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.290914 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.290866 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.290914 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.290917 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03088fcd-2439-4eca-bc07-c94f576a6e70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.291186 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.291017 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.291186 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.291094 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.291186 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.291129 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.291363 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.291193 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.291363 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.291239 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.291466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.291364 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.291466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.291399 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv68g\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-kube-api-access-nv68g\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.336069 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.336035 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6"] Nov 02 19:37:16.392840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.392790 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.392840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.392838 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03088fcd-2439-4eca-bc07-c94f576a6e70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393062 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.392947 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393062 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393031 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393060 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393088 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393114 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393230 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393258 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393304 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv68g\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-kube-api-access-nv68g\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393374 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393443 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393712 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393689 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.393779 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.393744 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03088fcd-2439-4eca-bc07-c94f576a6e70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.395297 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.395274 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.395632 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.395609 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.452419 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.452387 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.452582 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.452554 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv68g\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-kube-api-access-nv68g\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-n29w6\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.541460 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.541370 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:16.958255 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:37:16.958219 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03088fcd_2439_4eca_bc07_c94f576a6e70.slice/crio-5e713e27956b28be82c949b2a75dacd363d075e7a53c936127a178ff76b9d240 WatchSource:0}: Error finding container 5e713e27956b28be82c949b2a75dacd363d075e7a53c936127a178ff76b9d240: Status 404 returned error can't find the container with id 5e713e27956b28be82c949b2a75dacd363d075e7a53c936127a178ff76b9d240 Nov 02 19:37:16.959825 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.959802 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6"] Nov 02 19:37:16.962321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.962287 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:37:16.962420 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.962359 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:37:16.962420 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:16.962388 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:37:17.075894 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:17.075860 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" event={"ID":"03088fcd-2439-4eca-bc07-c94f576a6e70","Type":"ContainerStarted","Data":"dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f"} Nov 02 19:37:17.076080 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:17.075914 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" event={"ID":"03088fcd-2439-4eca-bc07-c94f576a6e70","Type":"ContainerStarted","Data":"5e713e27956b28be82c949b2a75dacd363d075e7a53c936127a178ff76b9d240"} Nov 02 19:37:17.198444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:17.198390 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" podStartSLOduration=1.1983755170000001 podStartE2EDuration="1.198375517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:37:17.196865995 +0000 UTC m=+495.539255075" watchObservedRunningTime="2026-11-02 19:37:17.198375517 +0000 UTC m=+495.540764594" Nov 02 19:37:17.541913 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:17.541878 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:18.543786 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:18.543750 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-n29w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 19:37:18.544196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:18.543803 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 19:37:19.542242 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:19.542202 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-n29w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 19:37:19.542419 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:19.542263 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 19:37:20.542772 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:20.542732 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-n29w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 19:37:20.543187 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:20.542807 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 19:37:21.541818 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:21.541782 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-n29w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 19:37:21.541990 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:21.541837 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 19:37:22.542132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:22.542090 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-n29w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 19:37:22.542586 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:22.542150 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 19:37:23.542590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:23.542499 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-n29w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 19:37:23.542590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:23.542555 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 19:37:24.886529 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:24.886467 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-fjcr4"] Nov 02 19:37:24.889343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:24.889322 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:24.904954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:24.904925 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:37:24.947058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:24.947017 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-fjcr4"] Nov 02 19:37:24.996617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:24.996584 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 19:37:24.996617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:24.996601 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 19:37:24.996617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:24.996601 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 19:37:24.996974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:24.996648 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-fwh4d\"" Nov 02 19:37:24.996974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:24.996674 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 19:37:25.002339 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.002273 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxkr\" (UniqueName: \"kubernetes.io/projected/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-kube-api-access-vlxkr\") pod \"ai-gateway-controller-55cd769764-fjcr4\" (UID: \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\") " pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:25.002548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.002512 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-metrics-certs\") pod \"ai-gateway-controller-55cd769764-fjcr4\" (UID: \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\") " pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:25.103585 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.103542 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-metrics-certs\") pod \"ai-gateway-controller-55cd769764-fjcr4\" (UID: \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\") " pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:25.103741 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.103717 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxkr\" (UniqueName: \"kubernetes.io/projected/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-kube-api-access-vlxkr\") pod \"ai-gateway-controller-55cd769764-fjcr4\" (UID: \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\") " pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:25.106079 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.106049 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-metrics-certs\") pod \"ai-gateway-controller-55cd769764-fjcr4\" (UID: \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\") " pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:25.163131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.163042 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxkr\" (UniqueName: \"kubernetes.io/projected/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-kube-api-access-vlxkr\") pod \"ai-gateway-controller-55cd769764-fjcr4\" (UID: \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\") " pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:25.168370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.168321 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:37:25.202220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.202182 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:25.334669 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.334633 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:25.335012 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.334992 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:25.335751 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.335737 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:37:25.607946 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:37:25.607892 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e3131b_eb23_4b93_9e43_aa2b7a1e3a44.slice/crio-2320a5efd0d7383d850dcbf2bc51627808b8561cb027eddd208fbb6074e489ea WatchSource:0}: Error finding container 2320a5efd0d7383d850dcbf2bc51627808b8561cb027eddd208fbb6074e489ea: Status 404 returned error can't find the container with id 2320a5efd0d7383d850dcbf2bc51627808b8561cb027eddd208fbb6074e489ea Nov 02 19:37:25.608797 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:25.608689 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-fjcr4"] Nov 02 19:37:26.118150 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:26.118118 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" event={"ID":"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44","Type":"ContainerStarted","Data":"2320a5efd0d7383d850dcbf2bc51627808b8561cb027eddd208fbb6074e489ea"} Nov 02 19:37:30.140055 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:30.140015 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" event={"ID":"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44","Type":"ContainerStarted","Data":"a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e"} Nov 02 19:37:30.140492 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:30.140114 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:30.257683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:30.257595 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" podStartSLOduration=2.5919819200000003 podStartE2EDuration="6.257581972s" totalImagesPullingTime="3.665600052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:37:30.25663749 +0000 UTC m=+508.599026571" watchObservedRunningTime="2026-11-02 19:37:30.257581972 +0000 UTC m=+508.599971052" Nov 02 19:37:33.166773 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.166741 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:37:33.334342 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.334296 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:37:33.545248 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.545171 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-m462h"] Nov 02 19:37:33.548152 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.548135 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:33.604279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.604209 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 19:37:33.656000 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.655954 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kq5qx\"" Nov 02 19:37:33.657361 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.657334 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-m462h"] Nov 02 19:37:33.699362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.699321 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5m9q\" (UniqueName: \"kubernetes.io/projected/8cc49add-9816-4d83-b8e9-e87fbedd5de2-kube-api-access-m5m9q\") pod \"payload-pre-processing-7b98bb678c-m462h\" (UID: \"8cc49add-9816-4d83-b8e9-e87fbedd5de2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:33.699615 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.699373 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cc49add-9816-4d83-b8e9-e87fbedd5de2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-m462h\" (UID: \"8cc49add-9816-4d83-b8e9-e87fbedd5de2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:33.801159 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.801051 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5m9q\" (UniqueName: \"kubernetes.io/projected/8cc49add-9816-4d83-b8e9-e87fbedd5de2-kube-api-access-m5m9q\") pod \"payload-pre-processing-7b98bb678c-m462h\" (UID: \"8cc49add-9816-4d83-b8e9-e87fbedd5de2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:33.801159 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.801134 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cc49add-9816-4d83-b8e9-e87fbedd5de2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-m462h\" (UID: \"8cc49add-9816-4d83-b8e9-e87fbedd5de2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:33.801878 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.801856 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cc49add-9816-4d83-b8e9-e87fbedd5de2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-m462h\" (UID: \"8cc49add-9816-4d83-b8e9-e87fbedd5de2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:33.860211 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.860166 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5m9q\" (UniqueName: \"kubernetes.io/projected/8cc49add-9816-4d83-b8e9-e87fbedd5de2-kube-api-access-m5m9q\") pod \"payload-pre-processing-7b98bb678c-m462h\" (UID: \"8cc49add-9816-4d83-b8e9-e87fbedd5de2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:33.860418 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:33.860378 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:34.229910 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:34.229881 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-m462h"] Nov 02 19:37:35.171609 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:35.171565 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" event={"ID":"8cc49add-9816-4d83-b8e9-e87fbedd5de2","Type":"ContainerStarted","Data":"cadadffcca2560090190b5bf001a64a83df6cea773ae02e9109451c8e1be6474"} Nov 02 19:37:38.187003 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:38.186961 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" event={"ID":"8cc49add-9816-4d83-b8e9-e87fbedd5de2","Type":"ContainerStarted","Data":"1854866ac1e1eb333447eedfafaa797bc7e93c611ef56849a0ef9e73b4355dd5"} Nov 02 19:37:38.187565 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:38.187104 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:38.303270 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:38.303166 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" podStartSLOduration=1.5773591900000001 podStartE2EDuration="5.30315226s" totalImagesPullingTime="3.72579307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:37:38.302580558 +0000 UTC m=+516.644969619" watchObservedRunningTime="2026-11-02 19:37:38.30315226 +0000 UTC m=+516.645541339" Nov 02 19:37:41.147500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.147443 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:41.501906 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.501825 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6c866c6d6d-md4dq"] Nov 02 19:37:41.505296 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.505277 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:41.564026 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.563996 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6c866c6d6d-md4dq"] Nov 02 19:37:41.588929 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.588893 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmtt\" (UniqueName: \"kubernetes.io/projected/940a998f-8cbe-42ac-ba71-4b36997feba2-kube-api-access-ghmtt\") pod \"ai-gateway-controller-6c866c6d6d-md4dq\" (UID: \"940a998f-8cbe-42ac-ba71-4b36997feba2\") " pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:41.588929 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.588933 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940a998f-8cbe-42ac-ba71-4b36997feba2-metrics-certs\") pod \"ai-gateway-controller-6c866c6d6d-md4dq\" (UID: \"940a998f-8cbe-42ac-ba71-4b36997feba2\") " pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:41.689404 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.689359 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmtt\" (UniqueName: \"kubernetes.io/projected/940a998f-8cbe-42ac-ba71-4b36997feba2-kube-api-access-ghmtt\") pod \"ai-gateway-controller-6c866c6d6d-md4dq\" (UID: \"940a998f-8cbe-42ac-ba71-4b36997feba2\") " pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:41.689404 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.689409 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940a998f-8cbe-42ac-ba71-4b36997feba2-metrics-certs\") pod \"ai-gateway-controller-6c866c6d6d-md4dq\" (UID: \"940a998f-8cbe-42ac-ba71-4b36997feba2\") " pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:41.691958 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.691933 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/940a998f-8cbe-42ac-ba71-4b36997feba2-metrics-certs\") pod \"ai-gateway-controller-6c866c6d6d-md4dq\" (UID: \"940a998f-8cbe-42ac-ba71-4b36997feba2\") " pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:41.748402 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.748370 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmtt\" (UniqueName: \"kubernetes.io/projected/940a998f-8cbe-42ac-ba71-4b36997feba2-kube-api-access-ghmtt\") pod \"ai-gateway-controller-6c866c6d6d-md4dq\" (UID: \"940a998f-8cbe-42ac-ba71-4b36997feba2\") " pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:41.817361 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:41.817284 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:42.189536 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:37:42.189499 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940a998f_8cbe_42ac_ba71_4b36997feba2.slice/crio-0f944ef62fcbc86cca0fed48e3c283e0e606d8aa8b8caf0305a224627ad7e4a3 WatchSource:0}: Error finding container 0f944ef62fcbc86cca0fed48e3c283e0e606d8aa8b8caf0305a224627ad7e4a3: Status 404 returned error can't find the container with id 0f944ef62fcbc86cca0fed48e3c283e0e606d8aa8b8caf0305a224627ad7e4a3 Nov 02 19:37:42.190172 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:42.190145 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6c866c6d6d-md4dq"] Nov 02 19:37:42.206487 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:42.206435 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" event={"ID":"940a998f-8cbe-42ac-ba71-4b36997feba2","Type":"ContainerStarted","Data":"0f944ef62fcbc86cca0fed48e3c283e0e606d8aa8b8caf0305a224627ad7e4a3"} Nov 02 19:37:43.213282 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:43.213238 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" event={"ID":"940a998f-8cbe-42ac-ba71-4b36997feba2","Type":"ContainerStarted","Data":"926c01017cf4ec1a8c9975a0a62d2db3d8e742ba524e7392249587a74a610b8a"} Nov 02 19:37:43.213697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:43.213388 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:43.330628 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:43.330520 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" podStartSLOduration=1.502368267 podStartE2EDuration="2.330506852s" totalImagesPullingTime="828.138585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:37:43.330227126 +0000 UTC m=+521.672616205" watchObservedRunningTime="2026-11-02 19:37:43.330506852 +0000 UTC m=+521.672895928" Nov 02 19:37:44.759896 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:37:44.759861 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:37:49.195439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:49.195410 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m462h" Nov 02 19:37:54.220341 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.220301 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6c866c6d6d-md4dq" Nov 02 19:37:54.391972 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.391934 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-fjcr4"] Nov 02 19:37:54.392228 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.392204 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" podUID="e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44" containerName="manager" containerID="cri-o://a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e" gracePeriod=30 Nov 02 19:37:54.638337 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.638311 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:54.719996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.719948 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-kube-api-access-vlxkr\" (UniqueName: \"kubernetes.io/projected/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-kube-api-access-vlxkr\") pod \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\" (UID: \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\") " Nov 02 19:37:54.720194 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.720038 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-metrics-certs\") pod \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\" (UID: \"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44\") " Nov 02 19:37:54.722205 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.722163 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-metrics-certs" pod "e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44" (UID: "e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:37:54.722303 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.722200 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-kube-api-access-vlxkr" pod "e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44" (UID: "e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44"). InnerVolumeSpecName "kube-api-access-vlxkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:37:54.821372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.821283 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlxkr\" (UniqueName: \"kubernetes.io/projected/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-kube-api-access-vlxkr\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:37:54.821372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:54.821318 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:37:55.269918 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.269881 5850 generic.go:356] "Generic (PLEG): container finished" podID="e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44" containerID="a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e" exitCode=0 Nov 02 19:37:55.269918 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.269922 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" event={"ID":"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44","Type":"ContainerDied","Data":"a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e"} Nov 02 19:37:55.270753 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.269929 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" Nov 02 19:37:55.270753 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.269965 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-fjcr4" event={"ID":"e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44","Type":"ContainerDied","Data":"2320a5efd0d7383d850dcbf2bc51627808b8561cb027eddd208fbb6074e489ea"} Nov 02 19:37:55.270753 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.269986 5850 scope.go:122] "RemoveContainer" containerID="a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e" Nov 02 19:37:55.278667 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.278639 5850 scope.go:122] "RemoveContainer" containerID="a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e" Nov 02 19:37:55.278946 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:37:55.278929 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e\": container with ID starting with a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e not found: ID does not exist" containerID="a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e" Nov 02 19:37:55.278991 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.278955 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e"} err="failed to get container status \"a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e\": rpc error: code = NotFound desc = could not find container \"a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e\": container with ID starting with a21e472c222d98620dd1ea62761f33168b83eb67810b0e3e51e5233bb54a4a2e not found: ID does not exist" Nov 02 19:37:55.437259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.437222 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-fjcr4"] Nov 02 19:37:55.441150 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.441122 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-fjcr4"] Nov 02 19:37:55.813952 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:55.813920 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44" path="/var/lib/kubelet/pods/e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44/volumes" Nov 02 19:37:56.054849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:56.054815 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:37:57.624751 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:57.624717 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:37:58.136100 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:58.136064 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:37:58.309269 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:37:58.309237 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:01.560225 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.560186 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t"] Nov 02 19:38:01.560865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.560700 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44" containerName="manager" Nov 02 19:38:01.560865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.560720 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44" containerName="manager" Nov 02 19:38:01.560865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.560818 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e3131b-eb23-4b93-9e43-aa2b7a1e3a44" containerName="manager" Nov 02 19:38:01.566463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.566440 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.667010 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.666972 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t"] Nov 02 19:38:01.693056 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.693019 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.693225 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.693060 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.693225 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.693081 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.693225 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.693199 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.693332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.693248 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.693332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.693317 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzcmt\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-kube-api-access-lzcmt\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.693393 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.693367 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.693393 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.693388 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3809068a-16bb-4072-9a21-b0b01d57ba93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.693483 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.693457 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.794359 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.794318 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzcmt\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-kube-api-access-lzcmt\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.794590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.794373 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.794590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.794406 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3809068a-16bb-4072-9a21-b0b01d57ba93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.794590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.794443 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.794590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.794516 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.794590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.794549 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.794590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.794586 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.794909 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.794726 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.794909 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.794808 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.795086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.795060 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.795147 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.795063 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.795147 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.795102 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.795241 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.795170 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.795329 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.795310 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3809068a-16bb-4072-9a21-b0b01d57ba93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.797048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.797025 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.797268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.797248 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.855730 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.855698 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.855937 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.855803 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzcmt\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-kube-api-access-lzcmt\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-xxp8t\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:01.879604 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:01.879574 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:02.300301 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:02.300264 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t"] Nov 02 19:38:02.336149 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:38:02.336117 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3809068a_16bb_4072_9a21_b0b01d57ba93.slice/crio-8efe428d746778d6b4d6e86d7d5178b840a054330fa58cea4e7a15c17b96b3c1 WatchSource:0}: Error finding container 8efe428d746778d6b4d6e86d7d5178b840a054330fa58cea4e7a15c17b96b3c1: Status 404 returned error can't find the container with id 8efe428d746778d6b4d6e86d7d5178b840a054330fa58cea4e7a15c17b96b3c1 Nov 02 19:38:02.338555 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:02.338520 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:38:02.338661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:02.338610 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:38:02.338728 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:02.338654 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:38:03.340399 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:03.340361 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" event={"ID":"3809068a-16bb-4072-9a21-b0b01d57ba93","Type":"ContainerStarted","Data":"a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5"} Nov 02 19:38:03.340849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:03.340442 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" event={"ID":"3809068a-16bb-4072-9a21-b0b01d57ba93","Type":"ContainerStarted","Data":"8efe428d746778d6b4d6e86d7d5178b840a054330fa58cea4e7a15c17b96b3c1"} Nov 02 19:38:03.519500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:03.517157 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" podStartSLOduration=2.517137397 podStartE2EDuration="2.517137397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:03.517059299 +0000 UTC m=+541.859448380" watchObservedRunningTime="2026-11-02 19:38:03.517137397 +0000 UTC m=+541.859526478" Nov 02 19:38:03.880548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:03.880499 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:03.882222 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:03.882194 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:38:03.882371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:03.882244 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 19:38:04.880381 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:04.880344 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:38:04.880804 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:04.880400 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 19:38:05.880292 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:05.880255 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:38:05.880468 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:05.880321 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 19:38:06.532946 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:06.532911 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:06.880717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:06.880687 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:38:06.881100 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:06.880755 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 19:38:07.880337 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:07.880301 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:38:07.880553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:07.880359 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 19:38:08.880078 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:08.880036 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 19:38:08.880508 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:08.880109 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 19:38:09.800201 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:09.800166 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:09.968015 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:09.967944 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:10.622772 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:10.622734 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:10.623347 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:10.623319 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:10.624115 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:10.624096 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:38:14.760822 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:38:14.760785 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:38:44.739186 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:44.739131 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:44.761681 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:38:44.761646 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:38:44.896972 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:44.896937 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:45.410674 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.410641 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:45.657143 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.657105 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25"] Nov 02 19:38:45.660781 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.660711 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.679551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.679523 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:45.717146 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.717105 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8795bf-openshift-default-dockercfg-s7q56\"" Nov 02 19:38:45.769978 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.769940 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25"] Nov 02 19:38:45.774861 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.774833 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.775001 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.774892 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.775088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.775062 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.775147 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.775112 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.775230 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.775211 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867b967-5ce9-4bca-908f-b9a9bffaa601-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.775266 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.775239 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.775266 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.775258 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.775356 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.775277 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.775356 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.775319 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q899p\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-kube-api-access-q899p\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.834826 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.834794 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:45.876953 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.876912 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.876953 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.876958 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.876988 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867b967-5ce9-4bca-908f-b9a9bffaa601-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877013 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877037 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877066 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877089 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q899p\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-kube-api-access-q899p\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877157 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877224 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877401 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877490 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877718 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877694 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877732 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.877785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.877754 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867b967-5ce9-4bca-908f-b9a9bffaa601-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.879729 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.879705 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.879906 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.879888 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.939540 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.939444 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.939685 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.939546 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q899p\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-kube-api-access-q899p\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.986120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.986082 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:45.998850 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:45.998812 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:46.164110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.164078 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:46.345122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.345035 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:46.464563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.464539 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25"] Nov 02 19:38:46.466078 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:38:46.466051 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0867b967_5ce9_4bca_908f_b9a9bffaa601.slice/crio-f39ef07075a0387f73137c6e8aefeff4a2366406fb3df74dc9cb2bc60c4c968e WatchSource:0}: Error finding container f39ef07075a0387f73137c6e8aefeff4a2366406fb3df74dc9cb2bc60c4c968e: Status 404 returned error can't find the container with id f39ef07075a0387f73137c6e8aefeff4a2366406fb3df74dc9cb2bc60c4c968e Nov 02 19:38:46.468179 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.468138 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:38:46.468243 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.468217 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:38:46.468278 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.468246 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:38:46.510836 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.510801 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:46.545090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.545050 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" event={"ID":"0867b967-5ce9-4bca-908f-b9a9bffaa601","Type":"ContainerStarted","Data":"5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c"} Nov 02 19:38:46.545245 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.545117 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" event={"ID":"0867b967-5ce9-4bca-908f-b9a9bffaa601","Type":"ContainerStarted","Data":"f39ef07075a0387f73137c6e8aefeff4a2366406fb3df74dc9cb2bc60c4c968e"} Nov 02 19:38:46.617263 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.617234 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls"] Nov 02 19:38:46.621080 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.621055 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.684654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.684602 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" podStartSLOduration=1.684584047 podStartE2EDuration="1.684584047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:46.666596498 +0000 UTC m=+585.008985570" watchObservedRunningTime="2026-11-02 19:38:46.684584047 +0000 UTC m=+585.026973175" Nov 02 19:38:46.685273 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.685245 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls"] Nov 02 19:38:46.686887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.686851 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.687111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.687090 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6clm\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-kube-api-access-s6clm\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.687279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.687257 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0c0942-7045-466b-88b2-a1a72f3afda0-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.687608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.687454 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.687608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.687504 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.687608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.687584 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.687608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.687613 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.687822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.687652 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.687822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.687679 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.688263 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.688236 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:46.733007 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.732957 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1d869e-openshift-default-dockercfg-cg57h\"" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.790871 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.790922 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.791033 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.791064 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.791112 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.791149 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.791199 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.791293 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6clm\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-kube-api-access-s6clm\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.791358 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0c0942-7045-466b-88b2-a1a72f3afda0-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.792006 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.792123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.792068 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.793238 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.792200 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0c0942-7045-466b-88b2-a1a72f3afda0-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.793238 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.792440 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.793238 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.792712 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.795608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.795578 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.807571 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.806661 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.850582 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.850541 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.850883 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.850858 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6clm\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-kube-api-access-s6clm\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.852535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.852513 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:46.935169 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.935081 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:46.992027 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:46.988148 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:47.027317 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.027283 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:47.207073 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.206983 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:47.359847 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:38:47.359813 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0c0942_7045_466b_88b2_a1a72f3afda0.slice/crio-20f4fe5d5c91367469de77c6c140f64e24c53a4c2ba569f5f8084b4d357fea50 WatchSource:0}: Error finding container 20f4fe5d5c91367469de77c6c140f64e24c53a4c2ba569f5f8084b4d357fea50: Status 404 returned error can't find the container with id 20f4fe5d5c91367469de77c6c140f64e24c53a4c2ba569f5f8084b4d357fea50 Nov 02 19:38:47.362095 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.362065 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls"] Nov 02 19:38:47.362457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.362421 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:38:47.362562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.362520 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:38:47.362688 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.362576 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:38:47.404245 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.404194 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:47.552365 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.552270 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" event={"ID":"9a0c0942-7045-466b-88b2-a1a72f3afda0","Type":"ContainerStarted","Data":"e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe"} Nov 02 19:38:47.552365 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.552334 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" event={"ID":"9a0c0942-7045-466b-88b2-a1a72f3afda0","Type":"ContainerStarted","Data":"20f4fe5d5c91367469de77c6c140f64e24c53a4c2ba569f5f8084b4d357fea50"} Nov 02 19:38:47.571865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.571838 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:47.672147 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.672088 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" podStartSLOduration=1.672073427 podStartE2EDuration="1.672073427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:47.670736229 +0000 UTC m=+586.013125308" watchObservedRunningTime="2026-11-02 19:38:47.672073427 +0000 UTC m=+586.014462501" Nov 02 19:38:47.735268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.735236 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:47.935210 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.935176 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:47.992414 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:47.992384 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:48.557921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:48.557890 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:48.558776 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:48.558759 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:38:48.940967 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:48.940939 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:49.341372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:49.341295 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:49.497894 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:49.497854 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:49.563824 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:49.563792 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:49.565397 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:49.565377 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:38:49.656049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:49.656019 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:49.822299 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:49.822261 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:50.518608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:50.518553 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:50.721019 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:50.720986 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:50.909338 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:50.909299 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:51.115015 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:51.114977 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:51.320578 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:51.320497 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:51.490399 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:51.490370 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:51.767750 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:51.767718 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:51.983153 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:51.983119 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:52.215745 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:52.215715 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:53.366058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:53.366016 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:53.526443 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:53.526408 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:53.689950 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:53.689915 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:53.959156 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:53.959079 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:54.146552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:54.146524 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:54.311397 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:54.311318 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:55.336566 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.336517 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl"] Nov 02 19:38:55.342522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.342489 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.388204 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.388168 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-metrics-certs\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.388385 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.388352 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.388438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.388381 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519e3ebf-d09c-45b5-a773-42b1dad0c314-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.388438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.388419 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vtj\" (UniqueName: \"kubernetes.io/projected/519e3ebf-d09c-45b5-a773-42b1dad0c314-kube-api-access-69vtj\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.398699 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.398658 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl"] Nov 02 19:38:55.448090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.448050 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-5c46af-serving-cert\"" Nov 02 19:38:55.448090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.448084 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sg6rf\"" Nov 02 19:38:55.448358 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.448154 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 19:38:55.489701 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.489668 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vtj\" (UniqueName: \"kubernetes.io/projected/519e3ebf-d09c-45b5-a773-42b1dad0c314-kube-api-access-69vtj\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.489870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.489718 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-metrics-certs\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.489870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.489809 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.489870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.489824 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519e3ebf-d09c-45b5-a773-42b1dad0c314-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.490032 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:38:55.489987 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-serving-cert: secret "maas-api-e2e-worker-w3-5c46af-serving-cert" not found Nov 02 19:38:55.490106 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:38:55.490092 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls podName:519e3ebf-d09c-45b5-a773-42b1dad0c314 nodeName:}" failed. No retries permitted until 2026-11-02 19:38:55.990067313 +0000 UTC m=+594.332456388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls") pod "maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" (UID: "519e3ebf-d09c-45b5-a773-42b1dad0c314") : secret "maas-api-e2e-worker-w3-5c46af-serving-cert" not found Nov 02 19:38:55.490415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.490397 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519e3ebf-d09c-45b5-a773-42b1dad0c314-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.492180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.492160 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-metrics-certs\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.535663 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.535628 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:55.558560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.558530 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vtj\" (UniqueName: \"kubernetes.io/projected/519e3ebf-d09c-45b5-a773-42b1dad0c314-kube-api-access-69vtj\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.691959 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.691925 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:55.875251 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.875216 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:55.995040 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.994956 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:55.997441 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:55.997409 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:56.257439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:56.257353 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:56.626726 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:38:56.626692 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519e3ebf_d09c_45b5_a773_42b1dad0c314.slice/crio-7ba9d00485eca54a66f3b51d4b6d27a6c7bda595bf226091e00a0bd38f47a966 WatchSource:0}: Error finding container 7ba9d00485eca54a66f3b51d4b6d27a6c7bda595bf226091e00a0bd38f47a966: Status 404 returned error can't find the container with id 7ba9d00485eca54a66f3b51d4b6d27a6c7bda595bf226091e00a0bd38f47a966 Nov 02 19:38:56.628407 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:56.628385 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl"] Nov 02 19:38:57.608077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:57.608041 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" event={"ID":"519e3ebf-d09c-45b5-a773-42b1dad0c314","Type":"ContainerStarted","Data":"2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739"} Nov 02 19:38:57.608248 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:57.608100 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" event={"ID":"519e3ebf-d09c-45b5-a773-42b1dad0c314","Type":"ContainerStarted","Data":"7ba9d00485eca54a66f3b51d4b6d27a6c7bda595bf226091e00a0bd38f47a966"} Nov 02 19:38:57.608248 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:57.608147 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:38:57.733953 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:57.733899 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" podStartSLOduration=1.8535699380000001 podStartE2EDuration="2.73388637s" totalImagesPullingTime="880.316432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:57.733381194 +0000 UTC m=+596.075770275" watchObservedRunningTime="2026-11-02 19:38:57.73388637 +0000 UTC m=+596.076275450" Nov 02 19:38:59.134086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.134050 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:59.317445 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.317409 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:38:59.674864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.674825 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx"] Nov 02 19:38:59.680005 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.679975 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:38:59.735832 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.735799 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx"] Nov 02 19:38:59.742438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.742408 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8pdm\" (UniqueName: \"kubernetes.io/projected/f8e19696-fe64-401f-874e-68e07b37b454-kube-api-access-r8pdm\") pod \"payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx\" (UID: \"f8e19696-fe64-401f-874e-68e07b37b454\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:38:59.742616 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.742461 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e19696-fe64-401f-874e-68e07b37b454-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx\" (UID: \"f8e19696-fe64-401f-874e-68e07b37b454\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:38:59.785095 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.785068 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5c46af-dockercfg-wr4v2\"" Nov 02 19:38:59.785095 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.785068 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5c46af\"" Nov 02 19:38:59.843914 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.843877 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pdm\" (UniqueName: \"kubernetes.io/projected/f8e19696-fe64-401f-874e-68e07b37b454-kube-api-access-r8pdm\") pod \"payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx\" (UID: \"f8e19696-fe64-401f-874e-68e07b37b454\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:38:59.844063 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.843930 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e19696-fe64-401f-874e-68e07b37b454-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx\" (UID: \"f8e19696-fe64-401f-874e-68e07b37b454\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:38:59.844660 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.844638 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e19696-fe64-401f-874e-68e07b37b454-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx\" (UID: \"f8e19696-fe64-401f-874e-68e07b37b454\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:38:59.902931 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.902896 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pdm\" (UniqueName: \"kubernetes.io/projected/f8e19696-fe64-401f-874e-68e07b37b454-kube-api-access-r8pdm\") pod \"payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx\" (UID: \"f8e19696-fe64-401f-874e-68e07b37b454\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:38:59.994096 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:38:59.994002 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:39:00.098067 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.098033 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84"] Nov 02 19:39:00.102868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.102841 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.160610 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.160576 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84"] Nov 02 19:39:00.202465 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.202439 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-43ceaa-serving-cert\"" Nov 02 19:39:00.248803 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.248714 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-metrics-certs\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.248803 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.248758 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.249019 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.248891 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp7d7\" (UniqueName: \"kubernetes.io/projected/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-kube-api-access-dp7d7\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.249019 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.248924 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.349587 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.349551 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-metrics-certs\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.349587 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.349591 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.349836 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.349654 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp7d7\" (UniqueName: \"kubernetes.io/projected/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-kube-api-access-dp7d7\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.349836 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.349677 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.349936 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:00.349843 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-serving-cert: secret "maas-api-e2e-worker-w0-43ceaa-serving-cert" not found Nov 02 19:39:00.349936 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:00.349919 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls podName:5aa037a9-534d-46b8-bebb-a3f2a7fa11c3 nodeName:}" failed. No retries permitted until 2026-11-02 19:39:00.849897693 +0000 UTC m=+599.192286771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls") pod "maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" (UID: "5aa037a9-534d-46b8-bebb-a3f2a7fa11c3") : secret "maas-api-e2e-worker-w0-43ceaa-serving-cert" not found Nov 02 19:39:00.350495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.350442 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.352283 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.352258 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-metrics-certs\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.366905 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:39:00.366872 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e19696_fe64_401f_874e_68e07b37b454.slice/crio-11ca2589410e9c35ceaf32cd98521707faef7ab313f08f359af818c43db577eb WatchSource:0}: Error finding container 11ca2589410e9c35ceaf32cd98521707faef7ab313f08f359af818c43db577eb: Status 404 returned error can't find the container with id 11ca2589410e9c35ceaf32cd98521707faef7ab313f08f359af818c43db577eb Nov 02 19:39:00.367936 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.367887 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:00.369859 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.369812 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx"] Nov 02 19:39:00.416706 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.416673 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp7d7\" (UniqueName: \"kubernetes.io/projected/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-kube-api-access-dp7d7\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.576386 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.575912 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:00.640419 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.640384 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" event={"ID":"f8e19696-fe64-401f-874e-68e07b37b454","Type":"ContainerStarted","Data":"f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9"} Nov 02 19:39:00.640623 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.640443 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" event={"ID":"f8e19696-fe64-401f-874e-68e07b37b454","Type":"ContainerStarted","Data":"11ca2589410e9c35ceaf32cd98521707faef7ab313f08f359af818c43db577eb"} Nov 02 19:39:00.640623 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.640489 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:39:00.725152 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.725120 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:00.757515 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.757450 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" podStartSLOduration=1.757437172 podStartE2EDuration="1.757437172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:00.756359181 +0000 UTC m=+599.098748262" watchObservedRunningTime="2026-11-02 19:39:00.757437172 +0000 UTC m=+599.099826252" Nov 02 19:39:00.854857 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.854770 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:00.857232 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:00.857200 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:01.016320 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:01.016280 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:01.435736 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:39:01.435702 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa037a9_534d_46b8_bebb_a3f2a7fa11c3.slice/crio-be13471ff3fb41a01149649d8ba57ba5ab30847b869c5ada97374abeb20d22d5 WatchSource:0}: Error finding container be13471ff3fb41a01149649d8ba57ba5ab30847b869c5ada97374abeb20d22d5: Status 404 returned error can't find the container with id be13471ff3fb41a01149649d8ba57ba5ab30847b869c5ada97374abeb20d22d5 Nov 02 19:39:01.436710 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:01.436682 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84"] Nov 02 19:39:01.646454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:01.646419 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" event={"ID":"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3","Type":"ContainerStarted","Data":"be13471ff3fb41a01149649d8ba57ba5ab30847b869c5ada97374abeb20d22d5"} Nov 02 19:39:01.784669 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:01.784586 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:01.966742 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:01.966708 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:02.653453 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:02.653397 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" event={"ID":"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3","Type":"ContainerStarted","Data":"84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e"} Nov 02 19:39:02.653908 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:02.653615 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:02.775271 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:02.775214 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" podStartSLOduration=1.8884724739999998 podStartE2EDuration="2.775200032s" totalImagesPullingTime="886.727558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:02.773726306 +0000 UTC m=+601.116115397" watchObservedRunningTime="2026-11-02 19:39:02.775200032 +0000 UTC m=+601.117589108" Nov 02 19:39:03.618594 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:03.618553 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 19:39:04.417922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:04.417894 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:04.575545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:04.575488 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:04.733066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:04.732960 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:04.903883 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:04.903829 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:05.088791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:05.088712 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:05.333973 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:05.333941 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:05.527817 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:05.527790 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:05.701265 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:05.701229 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:06.822290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:06.822253 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:08.664808 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:08.664778 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 19:39:09.616043 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:09.616012 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:09.797794 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:09.797761 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:09.988705 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:09.988669 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4"] Nov 02 19:39:09.992939 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:09.992918 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.048651 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.048619 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8795bf-serving-cert\"" Nov 02 19:39:10.102068 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.102035 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4"] Nov 02 19:39:10.167668 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.167634 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd60f-3fb8-47fa-a67a-3248d6849fed-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.167877 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.167683 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-metrics-certs\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.167877 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.167760 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h577\" (UniqueName: \"kubernetes.io/projected/309cd60f-3fb8-47fa-a67a-3248d6849fed-kube-api-access-6h577\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.168016 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.167915 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.192337 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.192296 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:10.268823 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.268743 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.268996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.268844 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd60f-3fb8-47fa-a67a-3248d6849fed-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.268996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.268885 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-metrics-certs\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.268996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.268903 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h577\" (UniqueName: \"kubernetes.io/projected/309cd60f-3fb8-47fa-a67a-3248d6849fed-kube-api-access-6h577\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.268996 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:10.268910 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-serving-cert: secret "maas-api-e2e-worker-w4-8795bf-serving-cert" not found Nov 02 19:39:10.269214 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:10.269002 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls podName:309cd60f-3fb8-47fa-a67a-3248d6849fed nodeName:}" failed. No retries permitted until 2026-11-02 19:39:10.768980051 +0000 UTC m=+609.111369131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls") pod "maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" (UID: "309cd60f-3fb8-47fa-a67a-3248d6849fed") : secret "maas-api-e2e-worker-w4-8795bf-serving-cert" not found Nov 02 19:39:10.269629 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.269605 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd60f-3fb8-47fa-a67a-3248d6849fed-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.271522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.271499 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-metrics-certs\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.327993 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.327960 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h577\" (UniqueName: \"kubernetes.io/projected/309cd60f-3fb8-47fa-a67a-3248d6849fed-kube-api-access-6h577\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.351309 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.351268 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:10.529807 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.529725 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:10.774371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.774336 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.776842 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.776814 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:10.906612 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:10.906573 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:11.294036 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:39:11.294004 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309cd60f_3fb8_47fa_a67a_3248d6849fed.slice/crio-153a52a8f42412c33b1eaecb7fdd2260fc5821d145486fc18b07d7532e40b16c WatchSource:0}: Error finding container 153a52a8f42412c33b1eaecb7fdd2260fc5821d145486fc18b07d7532e40b16c: Status 404 returned error can't find the container with id 153a52a8f42412c33b1eaecb7fdd2260fc5821d145486fc18b07d7532e40b16c Nov 02 19:39:11.296237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:11.296212 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4"] Nov 02 19:39:11.650115 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:11.650088 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 19:39:11.700263 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:11.700224 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" event={"ID":"309cd60f-3fb8-47fa-a67a-3248d6849fed","Type":"ContainerStarted","Data":"153a52a8f42412c33b1eaecb7fdd2260fc5821d145486fc18b07d7532e40b16c"} Nov 02 19:39:12.706587 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:12.706546 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" event={"ID":"309cd60f-3fb8-47fa-a67a-3248d6849fed","Type":"ContainerStarted","Data":"c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7"} Nov 02 19:39:12.707001 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:12.706720 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:14.764801 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:14.764760 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:39:14.773965 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:14.773939 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:14.973341 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:14.973304 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:15.122176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:15.122096 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:15.281028 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:15.280972 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:15.439633 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:15.439557 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:18.739686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:18.739607 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 19:39:18.858093 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:18.858038 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" podStartSLOduration=8.883243048 podStartE2EDuration="9.858023932s" totalImagesPullingTime="974.780884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:12.834869424 +0000 UTC m=+611.177258505" watchObservedRunningTime="2026-11-02 19:39:18.858023932 +0000 UTC m=+617.200413011" Nov 02 19:39:20.027324 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:20.027290 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:20.606666 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:20.606626 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:39:20.606845 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:20.606688 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:39:21.787603 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:21.787572 5850 generic.go:356] "Generic (PLEG): container finished" podID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerID="e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe" exitCode=0 Nov 02 19:39:21.788049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:21.787645 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" event={"ID":"9a0c0942-7045-466b-88b2-a1a72f3afda0","Type":"ContainerDied","Data":"e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe"} Nov 02 19:39:21.788049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:21.788054 5850 scope.go:122] "RemoveContainer" containerID="e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe" Nov 02 19:39:21.793083 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:21.793036 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:39:21.793431 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:21.793405 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:39:21.794036 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:21.793946 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:39:22.224957 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.224920 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:22.432655 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.432616 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:22.605895 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.605783 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:22.795368 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.795333 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" event={"ID":"9a0c0942-7045-466b-88b2-a1a72f3afda0","Type":"ContainerStarted","Data":"6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf"} Nov 02 19:39:22.825980 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.825933 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg"] Nov 02 19:39:22.830169 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.830136 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:22.867099 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.867068 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg"] Nov 02 19:39:22.901807 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.901777 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-28623b\"" Nov 02 19:39:22.902218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.902187 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-28623b-dockercfg-xvvkq\"" Nov 02 19:39:22.947883 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.947848 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:39:22.969533 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.968002 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 19:39:22.969533 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.968062 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 19:39:22.969883 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.969728 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg\" (UID: \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:22.969883 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:22.969848 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2td7\" (UniqueName: \"kubernetes.io/projected/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-kube-api-access-m2td7\") pod \"payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg\" (UID: \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:23.070695 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.070661 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg\" (UID: \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:23.070954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.070938 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2td7\" (UniqueName: \"kubernetes.io/projected/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-kube-api-access-m2td7\") pod \"payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg\" (UID: \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:23.071921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.071877 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg\" (UID: \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:23.157805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.157725 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2td7\" (UniqueName: \"kubernetes.io/projected/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-kube-api-access-m2td7\") pod \"payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg\" (UID: \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:23.163437 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.163211 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:23.580357 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:39:23.579301 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca4a5b1_35c5_4aa0_bb64_1f1f86f94c02.slice/crio-ec1f26a651a96606f4c49d38133d0e4dcce824db973ceb793d08f23905e4ea3e WatchSource:0}: Error finding container ec1f26a651a96606f4c49d38133d0e4dcce824db973ceb793d08f23905e4ea3e: Status 404 returned error can't find the container with id ec1f26a651a96606f4c49d38133d0e4dcce824db973ceb793d08f23905e4ea3e Nov 02 19:39:23.581132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.581106 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg"] Nov 02 19:39:23.802033 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.801993 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" event={"ID":"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02","Type":"ContainerStarted","Data":"fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed"} Nov 02 19:39:23.802465 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.802061 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" event={"ID":"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02","Type":"ContainerStarted","Data":"ec1f26a651a96606f4c49d38133d0e4dcce824db973ceb793d08f23905e4ea3e"} Nov 02 19:39:23.802679 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.802633 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:23.939496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.938867 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 19:39:23.939496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:23.938931 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 19:39:24.857419 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:24.857368 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:24.936362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:24.936327 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 19:39:24.936679 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:24.936649 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 19:39:25.939150 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:25.939110 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 19:39:25.939877 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:25.939843 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 19:39:26.943129 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:26.943089 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 19:39:26.943777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:26.943743 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 19:39:27.937660 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:27.937616 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 19:39:27.937897 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:27.937672 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 19:39:28.940120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:28.940085 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:39:28.940593 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:28.940368 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:39:28.941362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:28.941336 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:39:28.985823 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:28.985763 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" podStartSLOduration=6.985749274 podStartE2EDuration="6.985749274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:23.930843234 +0000 UTC m=+622.273232317" watchObservedRunningTime="2026-11-02 19:39:28.985749274 +0000 UTC m=+627.328138353" Nov 02 19:39:28.987455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:28.987430 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm"] Nov 02 19:39:28.990655 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:28.990638 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.046765 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.046722 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-e9c144-serving-cert\"" Nov 02 19:39:29.049748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.049714 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm"] Nov 02 19:39:29.073658 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.073616 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjghb\" (UniqueName: \"kubernetes.io/projected/82ae3868-9c4c-4c85-81ba-7633c542fa04-kube-api-access-hjghb\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.073866 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.073696 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-metrics-certs\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.073999 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.073967 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ae3868-9c4c-4c85-81ba-7633c542fa04-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.074396 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.074071 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.174947 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.174903 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-metrics-certs\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.175125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.175025 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ae3868-9c4c-4c85-81ba-7633c542fa04-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.175125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.175081 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.175230 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.175136 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjghb\" (UniqueName: \"kubernetes.io/projected/82ae3868-9c4c-4c85-81ba-7633c542fa04-kube-api-access-hjghb\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.175295 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:29.175253 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-serving-cert: secret "maas-api-e2e-worker-w6-e9c144-serving-cert" not found Nov 02 19:39:29.175343 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:29.175333 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls podName:82ae3868-9c4c-4c85-81ba-7633c542fa04 nodeName:}" failed. No retries permitted until 2026-11-02 19:39:29.675312311 +0000 UTC m=+628.017701388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls") pod "maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" (UID: "82ae3868-9c4c-4c85-81ba-7633c542fa04") : secret "maas-api-e2e-worker-w6-e9c144-serving-cert" not found Nov 02 19:39:29.175804 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.175780 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ae3868-9c4c-4c85-81ba-7633c542fa04-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.177768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.177751 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-metrics-certs\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.197358 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.197278 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:29.234554 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.234512 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjghb\" (UniqueName: \"kubernetes.io/projected/82ae3868-9c4c-4c85-81ba-7633c542fa04-kube-api-access-hjghb\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.365110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.365056 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:29.539725 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.539641 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:29.683634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.683585 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.686112 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.686078 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls\") pod \"maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:29.904885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:29.904843 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:30.277741 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:39:30.277701 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ae3868_9c4c_4c85_81ba_7633c542fa04.slice/crio-5e63c0b6f5ec0e988c4477ffdfa03a08773d5518291c6923401c752815c3f16e WatchSource:0}: Error finding container 5e63c0b6f5ec0e988c4477ffdfa03a08773d5518291c6923401c752815c3f16e: Status 404 returned error can't find the container with id 5e63c0b6f5ec0e988c4477ffdfa03a08773d5518291c6923401c752815c3f16e Nov 02 19:39:30.279917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:30.279886 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm"] Nov 02 19:39:30.420059 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:30.420008 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:30.874777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:30.874743 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" event={"ID":"82ae3868-9c4c-4c85-81ba-7633c542fa04","Type":"ContainerStarted","Data":"5e63c0b6f5ec0e988c4477ffdfa03a08773d5518291c6923401c752815c3f16e"} Nov 02 19:39:31.039304 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.039270 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:31.223996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.222584 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:31.587573 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:31.587530 5850 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0867b967_5ce9_4bca_908f_b9a9bffaa601.slice/crio-conmon-5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0867b967_5ce9_4bca_908f_b9a9bffaa601.slice/crio-5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c.scope\": RecentStats: unable to find data in memory cache]" Nov 02 19:39:31.588090 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:31.587596 5850 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0867b967_5ce9_4bca_908f_b9a9bffaa601.slice/crio-conmon-5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0867b967_5ce9_4bca_908f_b9a9bffaa601.slice/crio-5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c.scope\": RecentStats: unable to find data in memory cache]" Nov 02 19:39:31.880970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.880938 5850 generic.go:356] "Generic (PLEG): container finished" podID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerID="5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c" exitCode=0 Nov 02 19:39:31.881146 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.881013 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" event={"ID":"0867b967-5ce9-4bca-908f-b9a9bffaa601","Type":"ContainerDied","Data":"5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c"} Nov 02 19:39:31.881450 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.881431 5850 scope.go:122] "RemoveContainer" containerID="5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c" Nov 02 19:39:31.882754 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.882730 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" event={"ID":"82ae3868-9c4c-4c85-81ba-7633c542fa04","Type":"ContainerStarted","Data":"94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f"} Nov 02 19:39:31.882860 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.882786 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:31.882973 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.882943 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:39:31.883053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.883037 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:39:31.883118 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:31.883091 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:39:32.210642 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:32.210532 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" podStartSLOduration=3.412760081 podStartE2EDuration="4.210515185s" totalImagesPullingTime="797.755104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:32.20867037 +0000 UTC m=+630.551059487" watchObservedRunningTime="2026-11-02 19:39:32.210515185 +0000 UTC m=+630.552904267" Nov 02 19:39:32.890046 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:32.890011 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" event={"ID":"0867b967-5ce9-4bca-908f-b9a9bffaa601","Type":"ContainerStarted","Data":"3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67"} Nov 02 19:39:32.986555 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:32.986522 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:39:32.987791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:32.987759 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:39:32.987905 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:32.987820 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 19:39:33.558212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:33.558174 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:39:33.987562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:33.987525 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:39:33.987980 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:33.987585 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 19:39:34.361938 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:34.361851 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:34.553943 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:34.553909 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:34.703863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:34.703828 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:34.963833 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:34.963741 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:34.987397 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:34.987360 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:39:34.987626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:34.987429 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 19:39:35.837426 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:35.837396 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 19:39:35.987395 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:35.987362 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:39:35.987734 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:35.987415 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 19:39:36.987033 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:36.986996 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 19:39:36.987237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:36.987059 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 19:39:37.895668 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:37.895640 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 19:39:38.057415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:38.057383 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:39:38.059412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:38.059380 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:39:39.486402 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:39.486294 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:44.252868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:44.252818 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:44.760672 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:39:44.760630 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:39:45.948790 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:45.948755 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:46.547909 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:46.547873 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:46.739595 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:46.738520 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:49.072118 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:49.072089 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:53.848857 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:53.848820 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:55.503416 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:55.503381 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:56.101683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:56.101645 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:56.280575 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:56.280121 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:57.891847 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:57.891804 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:39:58.833523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:39:58.833457 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:01.342645 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:01.342609 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:01.925426 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:01.925398 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:02.101090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:02.101059 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:02.537077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:02.537040 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:02.711750 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:02.711708 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:03.581646 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:03.581610 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:07.145849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:07.145811 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:07.477377 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:07.477283 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:08.255412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:08.255380 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:12.731106 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:12.731072 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:14.761625 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:40:14.761581 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:40:16.692996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:16.692955 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:17.465573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:17.465529 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:17.688427 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:17.688394 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:18.202491 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:18.202435 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:18.365621 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:18.365579 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:22.247255 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:22.247221 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:22.875231 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:22.875194 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:23.562423 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:23.562326 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:23.737812 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:23.737743 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:23.919796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:23.919763 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:26.854808 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:26.854767 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:29.174852 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.174816 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5"] Nov 02 19:40:29.177857 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.177839 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.233830 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.233799 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"openshift-service-ca.crt\"" Nov 02 19:40:29.277890 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.277846 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-model-cache\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.277890 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.277887 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90f1e76-4c42-4056-85f2-b3502e2e4359-tls-certs\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.278133 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.277954 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-dshm\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.278133 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.278028 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-tmp-dir\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.278133 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.278066 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4ch\" (UniqueName: \"kubernetes.io/projected/d90f1e76-4c42-4056-85f2-b3502e2e4359-kube-api-access-fk4ch\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.278133 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.278086 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-home\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.283511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.283464 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:29.285196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.285174 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"kube-root-ca.crt\"" Nov 02 19:40:29.285297 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.285280 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"default-dockercfg-9c7rx\"" Nov 02 19:40:29.285391 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.285376 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"simulator-w5-1d869e-kserve-self-signed-certs\"" Nov 02 19:40:29.287418 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.287396 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5"] Nov 02 19:40:29.379214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.379180 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-model-cache\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.379214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.379217 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90f1e76-4c42-4056-85f2-b3502e2e4359-tls-certs\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.379455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.379253 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-dshm\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.379455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.379298 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-tmp-dir\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.379455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.379319 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4ch\" (UniqueName: \"kubernetes.io/projected/d90f1e76-4c42-4056-85f2-b3502e2e4359-kube-api-access-fk4ch\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.379455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.379342 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-home\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.379709 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.379683 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-model-cache\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.379774 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.379694 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-tmp-dir\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.379844 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.379816 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-home\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.381568 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.381551 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-dshm\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.381864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.381848 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90f1e76-4c42-4056-85f2-b3502e2e4359-tls-certs\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.439929 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.439852 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4ch\" (UniqueName: \"kubernetes.io/projected/d90f1e76-4c42-4056-85f2-b3502e2e4359-kube-api-access-fk4ch\") pod \"simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.494544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.494506 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:29.863422 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:40:29.863391 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd90f1e76_4c42_4056_85f2_b3502e2e4359.slice/crio-8fac629c4f286cdfc3ecc244eecd9f5518a080c7db2c543c1e82cd86d7a1c96d WatchSource:0}: Error finding container 8fac629c4f286cdfc3ecc244eecd9f5518a080c7db2c543c1e82cd86d7a1c96d: Status 404 returned error can't find the container with id 8fac629c4f286cdfc3ecc244eecd9f5518a080c7db2c543c1e82cd86d7a1c96d Nov 02 19:40:29.864690 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:29.864662 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5"] Nov 02 19:40:30.216439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:30.216401 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" event={"ID":"d90f1e76-4c42-4056-85f2-b3502e2e4359","Type":"ContainerStarted","Data":"e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548"} Nov 02 19:40:30.216905 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:30.216449 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" event={"ID":"d90f1e76-4c42-4056-85f2-b3502e2e4359","Type":"ContainerStarted","Data":"8fac629c4f286cdfc3ecc244eecd9f5518a080c7db2c543c1e82cd86d7a1c96d"} Nov 02 19:40:30.334659 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:30.334602 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podStartSLOduration=1.3345854369999999 podStartE2EDuration="1.334585437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:40:30.333278275 +0000 UTC m=+688.675667356" watchObservedRunningTime="2026-11-02 19:40:30.334585437 +0000 UTC m=+688.676974510" Nov 02 19:40:30.495130 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:30.495041 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:31.248322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:31.248284 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:35.642396 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:35.642357 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:36.418862 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:36.418825 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:39.495428 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:39.495389 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:39.508206 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:39.508176 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:40.142951 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:40.142916 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:40.278320 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:40.278284 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:40:44.290886 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.290846 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g"] Nov 02 19:40:44.294174 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.294152 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.345162 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.345126 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-home\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.345325 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.345173 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tls-certs\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.345396 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.345323 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tmp-dir\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.345440 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.345404 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt77m\" (UniqueName: \"kubernetes.io/projected/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-kube-api-access-xt77m\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.345440 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.345433 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-dshm\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.345524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.345485 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-model-cache\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.399380 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.399322 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:44.400432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.400414 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"kube-root-ca.crt\"" Nov 02 19:40:44.400600 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.400503 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"openshift-service-ca.crt\"" Nov 02 19:40:44.400600 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.400520 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"default-dockercfg-2bf9d\"" Nov 02 19:40:44.400600 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.400592 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"simulator-w6-e9c144-kserve-self-signed-certs\"" Nov 02 19:40:44.402424 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.402403 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g"] Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.453076 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tmp-dir\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.453130 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt77m\" (UniqueName: \"kubernetes.io/projected/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-kube-api-access-xt77m\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.453169 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-dshm\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.453230 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-model-cache\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.453321 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-home\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.453347 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tls-certs\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.454388 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-model-cache\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.454432 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-home\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.454714 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tmp-dir\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.456620 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-dshm\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.457343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.457227 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tls-certs\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.512659 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.512623 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt77m\" (UniqueName: \"kubernetes.io/projected/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-kube-api-access-xt77m\") pod \"simulator-w6-e9c144-kserve-775856685f-wj75g\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.561226 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.561126 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:44.605796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.605756 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:44.775198 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:40:44.775164 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:40:44.987796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:44.983648 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g"] Nov 02 19:40:45.293696 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:45.293607 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" event={"ID":"cdc04263-4ed8-4e57-893e-d6ce55d7cd53","Type":"ContainerStarted","Data":"63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c"} Nov 02 19:40:45.293696 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:45.293648 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" event={"ID":"cdc04263-4ed8-4e57-893e-d6ce55d7cd53","Type":"ContainerStarted","Data":"91f4b77c74f7735a8af3831d82e28db6d7552ee2a51e4339f86799306a1748f4"} Nov 02 19:40:45.416371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:45.416314 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podStartSLOduration=1.416300651 podStartE2EDuration="1.416300651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:40:45.415109928 +0000 UTC m=+703.757499008" watchObservedRunningTime="2026-11-02 19:40:45.416300651 +0000 UTC m=+703.758689730" Nov 02 19:40:45.606715 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:45.606623 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:48.958291 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:48.958257 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:53.339029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:53.338992 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:54.606661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:54.606620 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:54.619310 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:54.619283 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:55.355002 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:55.354965 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:40:56.942057 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:56.942021 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:40:57.750486 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:40:57.750450 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:02.133844 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:02.133808 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:06.323796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:06.323761 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:06.533707 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:06.533672 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:07.443629 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:07.443582 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:10.940785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:10.940697 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:14.761161 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:41:14.761117 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:41:15.426671 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:15.426635 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:19.859872 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:19.859834 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:23.124283 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.124247 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm"] Nov 02 19:41:23.128937 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.128917 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.185797 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.185762 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm"] Nov 02 19:41:23.186866 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.186846 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-home\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.186937 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.186885 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsmd\" (UniqueName: \"kubernetes.io/projected/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-kube-api-access-rtsmd\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.186937 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.186914 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-dshm\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.187081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.187054 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tls-certs\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.187201 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.187114 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tmp-dir\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.187248 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.187198 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-model-cache\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.230440 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.230409 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:23.234823 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.234791 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"openshift-service-ca.crt\"" Nov 02 19:41:23.234823 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.234775 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"default-dockercfg-h46nk\"" Nov 02 19:41:23.235027 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.234842 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"kube-root-ca.crt\"" Nov 02 19:41:23.235027 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.234958 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"premium-w4-8795bf-kserve-self-signed-certs\"" Nov 02 19:41:23.288169 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.288129 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-model-cache\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.289138 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.289011 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-model-cache\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.289675 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.289610 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-home\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.289816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.289719 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsmd\" (UniqueName: \"kubernetes.io/projected/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-kube-api-access-rtsmd\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.289816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.289760 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-dshm\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.289938 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.289819 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tls-certs\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.289938 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.289886 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tmp-dir\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.291456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.290307 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tmp-dir\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.291456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.290641 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-home\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.302156 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.302124 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-dshm\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.303077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.303045 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tls-certs\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.349370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.349339 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsmd\" (UniqueName: \"kubernetes.io/projected/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-kube-api-access-rtsmd\") pod \"premium-w4-8795bf-kserve-c6d7c7f4c-j92bm\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.441115 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.441007 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:23.838149 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:41:23.838115 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf446dc3b_8567_47c7_91b1_5291aa4fb4f0.slice/crio-0a8bfc0d635a8244d93a05e2e44d4b07914702acfa44d42c398a1771c5101184 WatchSource:0}: Error finding container 0a8bfc0d635a8244d93a05e2e44d4b07914702acfa44d42c398a1771c5101184: Status 404 returned error can't find the container with id 0a8bfc0d635a8244d93a05e2e44d4b07914702acfa44d42c398a1771c5101184 Nov 02 19:41:23.840298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:23.840278 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm"] Nov 02 19:41:24.234889 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:24.234853 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:24.518822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:24.518734 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" event={"ID":"f446dc3b-8567-47c7-91b1-5291aa4fb4f0","Type":"ContainerStarted","Data":"41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44"} Nov 02 19:41:24.518822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:24.518776 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" event={"ID":"f446dc3b-8567-47c7-91b1-5291aa4fb4f0","Type":"ContainerStarted","Data":"0a8bfc0d635a8244d93a05e2e44d4b07914702acfa44d42c398a1771c5101184"} Nov 02 19:41:24.636712 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:24.636656 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podStartSLOduration=1.636641834 podStartE2EDuration="1.636641834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:41:24.635251318 +0000 UTC m=+742.977640399" watchObservedRunningTime="2026-11-02 19:41:24.636641834 +0000 UTC m=+742.979030925" Nov 02 19:41:25.441796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:25.441753 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:28.588711 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:28.588679 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:33.027456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:33.027421 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:33.441573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:33.441536 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:33.454651 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:33.454621 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:33.578628 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:33.578590 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:41:37.437972 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:37.437936 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:41.850155 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:41.850122 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:44.761867 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:41:44.761821 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:41:46.225674 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:46.225627 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:46.389577 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:46.389543 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:50.790735 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:50.790696 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:55.227275 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:55.227228 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:41:59.642860 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:41:59.642820 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:01.748587 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:01.748552 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:04.066952 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:04.066918 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:07.767970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:07.767933 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:08.445793 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:08.445759 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:12.930050 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:12.930014 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:14.263167 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:14.263136 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:14.760901 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:42:14.760865 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:42:17.341082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:17.341041 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:21.043780 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:21.043743 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:21.773375 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:21.773343 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:26.237989 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:26.237953 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:30.739740 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:30.739695 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:32.692585 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.692551 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6"] Nov 02 19:42:32.696215 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.696198 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.750155 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.750102 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5c46af-openshift-default-dockercfg-c97j7\"" Nov 02 19:42:32.760632 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.760594 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.760800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.760654 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.760800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.760737 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.760800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.760780 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlb2h\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-kube-api-access-nlb2h\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.760946 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.760879 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533d8bea-d517-4550-b754-85bedfa0200d-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.760946 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.760920 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.760946 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.760942 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.761079 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.761009 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533d8bea-d517-4550-b754-85bedfa0200d-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.761079 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.761042 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.804116 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.804082 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6"] Nov 02 19:42:32.862336 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862299 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862345 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlb2h\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-kube-api-access-nlb2h\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862390 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533d8bea-d517-4550-b754-85bedfa0200d-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862418 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862437 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862468 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533d8bea-d517-4550-b754-85bedfa0200d-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862551 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862786 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862610 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862786 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862665 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862881 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862809 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862888 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.862996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.862972 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.863074 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.863052 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.863465 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.863441 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533d8bea-d517-4550-b754-85bedfa0200d-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.865278 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.865249 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.865493 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.865461 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533d8bea-d517-4550-b754-85bedfa0200d-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.920719 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.920685 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:32.920857 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:32.920767 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlb2h\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-kube-api-access-nlb2h\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:33.006945 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.006861 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm"] Nov 02 19:42:33.011919 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.011892 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.037978 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.037947 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:33.065665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.065625 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-model-cache\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.065827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.065677 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5664696d-245a-433b-8659-c44fd4a93e26-tls-certs\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.065827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.065712 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-home\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.065827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.065793 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-dshm\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.065827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.065823 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8xgq\" (UniqueName: \"kubernetes.io/projected/5664696d-245a-433b-8659-c44fd4a93e26-kube-api-access-h8xgq\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.066032 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.065871 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-tmp-dir\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.066899 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.066870 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"openshift-service-ca.crt\"" Nov 02 19:42:33.067043 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.067017 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"default-dockercfg-tjrwt\"" Nov 02 19:42:33.067207 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.067074 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"premium-w2-28623b-kserve-self-signed-certs\"" Nov 02 19:42:33.067317 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.067158 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"kube-root-ca.crt\"" Nov 02 19:42:33.069771 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.069739 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm"] Nov 02 19:42:33.122328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.122299 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:33.178888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.174534 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-dshm\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.178888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.174592 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8xgq\" (UniqueName: \"kubernetes.io/projected/5664696d-245a-433b-8659-c44fd4a93e26-kube-api-access-h8xgq\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.178888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.174639 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-tmp-dir\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.178888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.174777 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-model-cache\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.178888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.174799 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5664696d-245a-433b-8659-c44fd4a93e26-tls-certs\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.178888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.174828 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-home\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.178888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.175314 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-home\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.178888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.177214 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-tmp-dir\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.178888 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.177854 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-model-cache\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.181552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.181502 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5664696d-245a-433b-8659-c44fd4a93e26-tls-certs\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.183031 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.182957 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-dshm\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.236692 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.236661 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8xgq\" (UniqueName: \"kubernetes.io/projected/5664696d-245a-433b-8659-c44fd4a93e26-kube-api-access-h8xgq\") pod \"premium-w2-28623b-kserve-5ccbbf64c7-9gklm\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.324898 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.324814 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:33.466345 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:42:33.466302 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533d8bea_d517_4550_b754_85bedfa0200d.slice/crio-d8dc38e9b61688051f8f45c683b22db7707bcf15b96326d2b920101023f2cc1a WatchSource:0}: Error finding container d8dc38e9b61688051f8f45c683b22db7707bcf15b96326d2b920101023f2cc1a: Status 404 returned error can't find the container with id d8dc38e9b61688051f8f45c683b22db7707bcf15b96326d2b920101023f2cc1a Nov 02 19:42:33.468419 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.468398 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6"] Nov 02 19:42:33.469170 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.469140 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:42:33.469232 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.469205 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:42:33.469269 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.469235 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:42:33.725521 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:42:33.724162 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5664696d_245a_433b_8659_c44fd4a93e26.slice/crio-07f292ae599191afc15c7ab5036afce0fbf8873c5230e44120a44ffab7454f40 WatchSource:0}: Error finding container 07f292ae599191afc15c7ab5036afce0fbf8873c5230e44120a44ffab7454f40: Status 404 returned error can't find the container with id 07f292ae599191afc15c7ab5036afce0fbf8873c5230e44120a44ffab7454f40 Nov 02 19:42:33.725521 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.725392 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm"] Nov 02 19:42:33.895664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.895017 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" event={"ID":"5664696d-245a-433b-8659-c44fd4a93e26","Type":"ContainerStarted","Data":"5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2"} Nov 02 19:42:33.896782 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.896745 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" event={"ID":"5664696d-245a-433b-8659-c44fd4a93e26","Type":"ContainerStarted","Data":"07f292ae599191afc15c7ab5036afce0fbf8873c5230e44120a44ffab7454f40"} Nov 02 19:42:33.896915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.896883 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" event={"ID":"533d8bea-d517-4550-b754-85bedfa0200d","Type":"ContainerStarted","Data":"d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9"} Nov 02 19:42:33.897157 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:33.897140 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" event={"ID":"533d8bea-d517-4550-b754-85bedfa0200d","Type":"ContainerStarted","Data":"d8dc38e9b61688051f8f45c683b22db7707bcf15b96326d2b920101023f2cc1a"} Nov 02 19:42:34.033907 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:34.023562 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podStartSLOduration=2.023543981 podStartE2EDuration="2.023543981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:42:34.021891525 +0000 UTC m=+812.364280606" watchObservedRunningTime="2026-11-02 19:42:34.023543981 +0000 UTC m=+812.365933065" Nov 02 19:42:34.038897 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:34.038862 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:34.040356 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:34.040325 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 19:42:34.040502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:34.040390 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 19:42:34.216515 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:34.213090 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:34.331595 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:34.327577 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:34.366695 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:34.366658 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:35.038466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:35.038426 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 19:42:35.039011 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:35.038511 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 19:42:35.158954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:35.158919 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:36.039030 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:36.038995 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 19:42:36.039430 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:36.039054 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 19:42:37.039115 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:37.039080 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 19:42:37.039643 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:37.039141 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 19:42:38.039106 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:38.039065 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 19:42:38.039305 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:38.039124 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 19:42:39.038887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:39.038850 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 19:42:39.039149 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:39.038907 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 19:42:39.526110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:39.526071 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:40.169218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:40.169184 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:40.170265 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:40.170247 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:40.170904 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:40.170888 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 19:42:40.341831 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:40.341776 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" podStartSLOduration=8.341762078 podStartE2EDuration="8.341762078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:42:34.149848628 +0000 UTC m=+812.492237710" watchObservedRunningTime="2026-11-02 19:42:40.341762078 +0000 UTC m=+818.684151157" Nov 02 19:42:43.325319 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:43.325270 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:43.337960 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:43.337935 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:44.025465 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:44.025437 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:42:44.046905 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:44.046818 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:44.761565 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:42:44.761528 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:42:48.565055 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.565018 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:48.808759 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.808726 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v"] Nov 02 19:42:48.812647 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.812631 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.871292 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.871260 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v"] Nov 02 19:42:48.900679 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.900639 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.900830 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.900694 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.900873 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.900830 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlnlx\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-kube-api-access-nlnlx\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.900873 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.900862 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.900989 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.900968 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.901045 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.901030 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.901151 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.901132 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.901215 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.901183 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.901319 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.901303 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:48.920583 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:48.920554 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e9c144-openshift-default-dockercfg-7slsb\"" Nov 02 19:42:49.002289 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002251 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.002464 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002334 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.002464 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002369 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.002464 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002405 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.002683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002522 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.002683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002563 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.002683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002588 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.002683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002654 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlnlx\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-kube-api-access-nlnlx\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.002977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002682 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.002977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.002879 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.003103 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.003004 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.003103 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.003047 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.003342 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.003319 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.003423 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.003327 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.005044 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.005020 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.005289 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.005273 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.061306 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.061271 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.061446 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.061408 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlnlx\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-kube-api-access-nlnlx\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.125640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.125610 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:49.546962 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:42:49.546928 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3f4c78_7fcb_4ae0_91fb_a548e58c2c4a.slice/crio-7341ee4adde783eb212b8ffd67490f9198fdcb6b1c6164af7a93bb82ab9f39f8 WatchSource:0}: Error finding container 7341ee4adde783eb212b8ffd67490f9198fdcb6b1c6164af7a93bb82ab9f39f8: Status 404 returned error can't find the container with id 7341ee4adde783eb212b8ffd67490f9198fdcb6b1c6164af7a93bb82ab9f39f8 Nov 02 19:42:49.549296 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.549268 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v"] Nov 02 19:42:49.549662 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.549628 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:42:49.549746 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.549707 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:42:49.549803 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:49.549754 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:42:50.046944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:50.046901 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" event={"ID":"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a","Type":"ContainerStarted","Data":"43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72"} Nov 02 19:42:50.047352 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:50.046968 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" event={"ID":"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a","Type":"ContainerStarted","Data":"7341ee4adde783eb212b8ffd67490f9198fdcb6b1c6164af7a93bb82ab9f39f8"} Nov 02 19:42:50.129496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:50.128667 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:50.131650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:50.131613 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 19:42:50.131816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:50.131677 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 19:42:50.222956 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:50.222881 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" podStartSLOduration=2.222861387 podStartE2EDuration="2.222861387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:42:50.165955648 +0000 UTC m=+828.508344725" watchObservedRunningTime="2026-11-02 19:42:50.222861387 +0000 UTC m=+828.565250471" Nov 02 19:42:51.132051 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:51.132004 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 19:42:51.132542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:51.132079 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 19:42:52.131974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:52.131931 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 19:42:52.132150 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:52.132019 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 19:42:52.952235 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:52.952192 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:42:53.127656 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:53.127613 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 19:42:53.127944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:53.127913 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 19:42:54.127983 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:54.127939 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 19:42:54.128644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:54.128611 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 19:42:55.128500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:55.127942 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 19:42:55.128500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:55.128023 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 19:42:56.260264 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:56.260231 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:56.260906 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:56.260882 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:56.261809 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:56.261786 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 19:42:57.346963 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:42:57.346919 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:01.838676 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:01.838638 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:03.697613 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:03.697577 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t"] Nov 02 19:43:03.698054 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:03.697842 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" containerID="cri-o://a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5" gracePeriod=30 Nov 02 19:43:03.748857 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:03.748823 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6"] Nov 02 19:43:03.749036 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:03.749010 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" containerID="cri-o://dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f" gracePeriod=30 Nov 02 19:43:06.261846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:06.261804 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:08.090489 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:08.090435 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:08.423157 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:08.423125 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:09.010899 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.010871 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:43:09.042020 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.041996 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:43:09.065904 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.065867 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-podinfo\") pod \"3809068a-16bb-4072-9a21-b0b01d57ba93\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " Nov 02 19:43:09.066048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.065913 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-kube-api-access-nv68g\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-kube-api-access-nv68g\") pod \"03088fcd-2439-4eca-bc07-c94f576a6e70\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " Nov 02 19:43:09.066048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.065966 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-data\") pod \"3809068a-16bb-4072-9a21-b0b01d57ba93\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " Nov 02 19:43:09.066048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.065998 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-credential-socket\") pod \"03088fcd-2439-4eca-bc07-c94f576a6e70\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " Nov 02 19:43:09.066197 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066048 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3809068a-16bb-4072-9a21-b0b01d57ba93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3809068a-16bb-4072-9a21-b0b01d57ba93-istiod-ca-cert\") pod \"3809068a-16bb-4072-9a21-b0b01d57ba93\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " Nov 02 19:43:09.066197 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066080 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-certs\") pod \"3809068a-16bb-4072-9a21-b0b01d57ba93\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " Nov 02 19:43:09.066197 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066118 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03088fcd-2439-4eca-bc07-c94f576a6e70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03088fcd-2439-4eca-bc07-c94f576a6e70-istiod-ca-cert\") pod \"03088fcd-2439-4eca-bc07-c94f576a6e70\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " Nov 02 19:43:09.066197 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066145 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-kube-api-access-lzcmt\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-kube-api-access-lzcmt\") pod \"3809068a-16bb-4072-9a21-b0b01d57ba93\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " Nov 02 19:43:09.066197 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066177 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-token\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-token\") pod \"03088fcd-2439-4eca-bc07-c94f576a6e70\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " Nov 02 19:43:09.066457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066210 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-envoy\") pod \"03088fcd-2439-4eca-bc07-c94f576a6e70\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " Nov 02 19:43:09.066457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066239 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-token\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-token\") pod \"3809068a-16bb-4072-9a21-b0b01d57ba93\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " Nov 02 19:43:09.066457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066318 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-envoy\") pod \"3809068a-16bb-4072-9a21-b0b01d57ba93\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " Nov 02 19:43:09.066457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066377 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-data\") pod \"03088fcd-2439-4eca-bc07-c94f576a6e70\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " Nov 02 19:43:09.067049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066311 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-data" pod "3809068a-16bb-4072-9a21-b0b01d57ba93" (UID: "3809068a-16bb-4072-9a21-b0b01d57ba93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.067049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066345 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-credential-socket" pod "03088fcd-2439-4eca-bc07-c94f576a6e70" (UID: "03088fcd-2439-4eca-bc07-c94f576a6e70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.067049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066641 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3809068a-16bb-4072-9a21-b0b01d57ba93-istiod-ca-cert" pod "3809068a-16bb-4072-9a21-b0b01d57ba93" (UID: "3809068a-16bb-4072-9a21-b0b01d57ba93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:09.067049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066670 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-data" pod "03088fcd-2439-4eca-bc07-c94f576a6e70" (UID: "03088fcd-2439-4eca-bc07-c94f576a6e70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.067049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066703 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03088fcd-2439-4eca-bc07-c94f576a6e70-istiod-ca-cert" pod "03088fcd-2439-4eca-bc07-c94f576a6e70" (UID: "03088fcd-2439-4eca-bc07-c94f576a6e70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:09.067049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066907 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-certs" pod "3809068a-16bb-4072-9a21-b0b01d57ba93" (UID: "3809068a-16bb-4072-9a21-b0b01d57ba93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.067049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.066979 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-credential-socket\") pod \"3809068a-16bb-4072-9a21-b0b01d57ba93\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " Nov 02 19:43:09.067524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067070 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-certs\") pod \"03088fcd-2439-4eca-bc07-c94f576a6e70\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " Nov 02 19:43:09.067524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067103 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-socket\") pod \"03088fcd-2439-4eca-bc07-c94f576a6e70\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " Nov 02 19:43:09.067524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067162 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-socket\") pod \"3809068a-16bb-4072-9a21-b0b01d57ba93\" (UID: \"3809068a-16bb-4072-9a21-b0b01d57ba93\") " Nov 02 19:43:09.067524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067243 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-podinfo\") pod \"03088fcd-2439-4eca-bc07-c94f576a6e70\" (UID: \"03088fcd-2439-4eca-bc07-c94f576a6e70\") " Nov 02 19:43:09.067808 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067783 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.067867 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067817 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.067867 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067833 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3809068a-16bb-4072-9a21-b0b01d57ba93-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.067867 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067847 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.067867 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067861 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03088fcd-2439-4eca-bc07-c94f576a6e70-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.068074 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.067876 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.068509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.068410 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-certs" pod "03088fcd-2439-4eca-bc07-c94f576a6e70" (UID: "03088fcd-2439-4eca-bc07-c94f576a6e70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.068606 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.068545 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-socket" pod "03088fcd-2439-4eca-bc07-c94f576a6e70" (UID: "03088fcd-2439-4eca-bc07-c94f576a6e70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.068667 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.068639 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-credential-socket" pod "3809068a-16bb-4072-9a21-b0b01d57ba93" (UID: "3809068a-16bb-4072-9a21-b0b01d57ba93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.068725 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.068699 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-socket" pod "3809068a-16bb-4072-9a21-b0b01d57ba93" (UID: "3809068a-16bb-4072-9a21-b0b01d57ba93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.068973 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.068929 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-token" pod "03088fcd-2439-4eca-bc07-c94f576a6e70" (UID: "03088fcd-2439-4eca-bc07-c94f576a6e70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:09.068973 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.068953 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-kube-api-access-nv68g" pod "03088fcd-2439-4eca-bc07-c94f576a6e70" (UID: "03088fcd-2439-4eca-bc07-c94f576a6e70"). InnerVolumeSpecName "kube-api-access-nv68g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:09.069165 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.069135 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-envoy" pod "3809068a-16bb-4072-9a21-b0b01d57ba93" (UID: "3809068a-16bb-4072-9a21-b0b01d57ba93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.069261 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.069174 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-envoy" pod "03088fcd-2439-4eca-bc07-c94f576a6e70" (UID: "03088fcd-2439-4eca-bc07-c94f576a6e70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:43:09.069261 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.069196 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-token" pod "3809068a-16bb-4072-9a21-b0b01d57ba93" (UID: "3809068a-16bb-4072-9a21-b0b01d57ba93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:09.069344 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.069327 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-podinfo" pod "3809068a-16bb-4072-9a21-b0b01d57ba93" (UID: "3809068a-16bb-4072-9a21-b0b01d57ba93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:43:09.070010 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.069991 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-kube-api-access-lzcmt" pod "3809068a-16bb-4072-9a21-b0b01d57ba93" (UID: "3809068a-16bb-4072-9a21-b0b01d57ba93"). InnerVolumeSpecName "kube-api-access-lzcmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:09.070303 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.070290 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-podinfo" pod "03088fcd-2439-4eca-bc07-c94f576a6e70" (UID: "03088fcd-2439-4eca-bc07-c94f576a6e70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:43:09.158443 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.158401 5850 generic.go:356] "Generic (PLEG): container finished" podID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerID="a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5" exitCode=0 Nov 02 19:43:09.158941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.158458 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" Nov 02 19:43:09.158941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.158503 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" event={"ID":"3809068a-16bb-4072-9a21-b0b01d57ba93","Type":"ContainerDied","Data":"a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5"} Nov 02 19:43:09.158941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.158565 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t" event={"ID":"3809068a-16bb-4072-9a21-b0b01d57ba93","Type":"ContainerDied","Data":"8efe428d746778d6b4d6e86d7d5178b840a054330fa58cea4e7a15c17b96b3c1"} Nov 02 19:43:09.158941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.158587 5850 scope.go:122] "RemoveContainer" containerID="a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5" Nov 02 19:43:09.159966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.159944 5850 generic.go:356] "Generic (PLEG): container finished" podID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerID="dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f" exitCode=0 Nov 02 19:43:09.160086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.159972 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" event={"ID":"03088fcd-2439-4eca-bc07-c94f576a6e70","Type":"ContainerDied","Data":"dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f"} Nov 02 19:43:09.160086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.159997 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" Nov 02 19:43:09.160086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.160023 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6" event={"ID":"03088fcd-2439-4eca-bc07-c94f576a6e70","Type":"ContainerDied","Data":"5e713e27956b28be82c949b2a75dacd363d075e7a53c936127a178ff76b9d240"} Nov 02 19:43:09.167915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.167895 5850 scope.go:122] "RemoveContainer" containerID="a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5" Nov 02 19:43:09.168284 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:43:09.168257 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5\": container with ID starting with a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5 not found: ID does not exist" containerID="a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5" Nov 02 19:43:09.168362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168294 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5"} err="failed to get container status \"a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5\": rpc error: code = NotFound desc = could not find container \"a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5\": container with ID starting with a1833d0a3eeaef519e243f7a670a4208312715d321f5e129558f4c094e1b4ce5 not found: ID does not exist" Nov 02 19:43:09.168362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168320 5850 scope.go:122] "RemoveContainer" containerID="dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f" Nov 02 19:43:09.168808 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168787 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168808 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168808 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168818 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv68g\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-kube-api-access-nv68g\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168827 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzcmt\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-kube-api-access-lzcmt\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168838 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168846 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168854 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168862 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168875 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168889 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168901 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03088fcd-2439-4eca-bc07-c94f576a6e70-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.168970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.168914 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3809068a-16bb-4072-9a21-b0b01d57ba93-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:09.182564 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.182530 5850 scope.go:122] "RemoveContainer" containerID="dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f" Nov 02 19:43:09.183410 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:43:09.183382 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f\": container with ID starting with dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f not found: ID does not exist" containerID="dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f" Nov 02 19:43:09.183534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.183417 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f"} err="failed to get container status \"dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f\": rpc error: code = NotFound desc = could not find container \"dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f\": container with ID starting with dea5bab933578d218aa12a203182604fb38e6f921613be2b12395c753bd7348f not found: ID does not exist" Nov 02 19:43:09.361824 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.361730 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6"] Nov 02 19:43:09.365516 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.365453 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-n29w6"] Nov 02 19:43:09.478611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.478564 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t"] Nov 02 19:43:09.481188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.481161 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xxp8t"] Nov 02 19:43:09.814519 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.814463 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" path="/var/lib/kubelet/pods/03088fcd-2439-4eca-bc07-c94f576a6e70/volumes" Nov 02 19:43:09.814984 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:09.814967 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" path="/var/lib/kubelet/pods/3809068a-16bb-4072-9a21-b0b01d57ba93/volumes" Nov 02 19:43:10.763950 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:10.763913 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:14.761749 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:43:14.761712 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:43:15.242350 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:15.242319 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:18.359089 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.359055 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27"] Nov 02 19:43:18.359545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.359530 5850 cpu_manager.go:414] "containerMap: removing container" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" Nov 02 19:43:18.359594 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.359546 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" Nov 02 19:43:18.359594 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.359561 5850 cpu_manager.go:414] "containerMap: removing container" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" Nov 02 19:43:18.359594 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.359566 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" Nov 02 19:43:18.359691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.359655 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="03088fcd-2439-4eca-bc07-c94f576a6e70" containerName="istio-proxy" Nov 02 19:43:18.359691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.359664 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="3809068a-16bb-4072-9a21-b0b01d57ba93" containerName="istio-proxy" Nov 02 19:43:18.362755 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.362737 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.419653 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.419616 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-26cfbe-openshift-default-dockercfg-msctx\"" Nov 02 19:43:18.471287 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.471249 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.471462 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.471382 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.471530 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.471470 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.471571 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.471552 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.471671 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.471650 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrl2j\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-kube-api-access-lrl2j\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.471709 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.471687 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.471743 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.471721 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b910d3-426d-4bff-8baf-7db10fb0937d-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.471831 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.471818 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.471865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.471841 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.473090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.473072 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27"] Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574101 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574203 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574256 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574310 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574371 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrl2j\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-kube-api-access-lrl2j\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574405 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574444 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b910d3-426d-4bff-8baf-7db10fb0937d-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574558 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574597 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.575389 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.574829 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.583589 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.577594 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.583589 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.577858 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.583589 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.578458 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b910d3-426d-4bff-8baf-7db10fb0937d-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.583589 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.578735 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.583589 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.578965 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.585704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.585676 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.634604 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.634569 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrl2j\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-kube-api-access-lrl2j\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.639505 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.639466 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:18.676196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:18.676156 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:19.097294 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:43:19.097265 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b910d3_426d_4bff_8baf_7db10fb0937d.slice/crio-7b873e4d2a7f36176f22c822c09d5c61e5ea84232f500f0b33e17c1819f7352b WatchSource:0}: Error finding container 7b873e4d2a7f36176f22c822c09d5c61e5ea84232f500f0b33e17c1819f7352b: Status 404 returned error can't find the container with id 7b873e4d2a7f36176f22c822c09d5c61e5ea84232f500f0b33e17c1819f7352b Nov 02 19:43:19.099670 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:19.099648 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27"] Nov 02 19:43:19.099760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:19.099649 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:43:19.099760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:19.099715 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:43:19.099760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:19.099747 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:43:19.220307 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:19.220271 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" event={"ID":"83b910d3-426d-4bff-8baf-7db10fb0937d","Type":"ContainerStarted","Data":"50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063"} Nov 02 19:43:19.220510 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:19.220321 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" event={"ID":"83b910d3-426d-4bff-8baf-7db10fb0937d","Type":"ContainerStarted","Data":"7b873e4d2a7f36176f22c822c09d5c61e5ea84232f500f0b33e17c1819f7352b"} Nov 02 19:43:19.339759 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:19.339702 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" podStartSLOduration=1.339680531 podStartE2EDuration="1.339680531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:19.338441442 +0000 UTC m=+857.680830540" watchObservedRunningTime="2026-11-02 19:43:19.339680531 +0000 UTC m=+857.682069613" Nov 02 19:43:19.669227 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:19.669186 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:19.677040 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:19.677010 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:20.678100 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:20.678064 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:43:20.678618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:20.678127 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 19:43:21.677139 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:21.677101 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:43:21.677399 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:21.677162 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 19:43:22.677425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:22.677384 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:43:22.677881 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:22.677445 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 19:43:23.676889 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:23.676856 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:43:23.677080 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:23.676908 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 19:43:24.158808 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:24.158759 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:24.676945 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:24.676909 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:43:24.677127 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:24.676978 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 19:43:25.802161 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:25.802112 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:25.803356 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:25.803329 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:25.804089 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:25.804065 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 19:43:28.563388 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:28.563352 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:32.949713 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:32.949682 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:37.454864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:37.454826 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:38.729215 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:38.729183 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:41.924132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:41.924094 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:44.760935 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:43:44.760892 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:43:46.335235 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:46.335197 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:49.732700 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:49.732667 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:50.732015 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:50.731985 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:55.163298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:55.163258 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:43:59.537345 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:43:59.537306 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:03.960677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:03.960641 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:08.385861 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:08.385825 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:08.544791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:08.544761 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:12.738501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:12.738392 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:14.761801 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:44:14.761759 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:44:17.097810 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.097777 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:17.787231 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.787197 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb"] Nov 02 19:44:17.791079 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.791058 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:17.845124 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.845090 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-43ceaa-openshift-default-dockercfg-nlhql\"" Nov 02 19:44:17.902030 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.901994 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb"] Nov 02 19:44:17.903967 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.903941 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64524fe5-c084-40bd-9195-7ca010e99c99-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:17.904117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.903982 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:17.904190 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.904149 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:17.904248 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.904214 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64524fe5-c084-40bd-9195-7ca010e99c99-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:17.904294 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.904253 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:17.904349 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.904323 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:17.904398 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.904374 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:17.904437 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.904412 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82m9\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-kube-api-access-l82m9\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:17.904581 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:17.904550 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.005416 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.005381 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64524fe5-c084-40bd-9195-7ca010e99c99-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.005662 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.005443 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.005662 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.005517 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.005662 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.005562 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64524fe5-c084-40bd-9195-7ca010e99c99-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.005662 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.005591 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.005662 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.005624 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.005662 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.005654 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.005970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.005676 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82m9\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-kube-api-access-l82m9\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.005970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.005713 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.006096 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.006058 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.006157 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.006074 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.006157 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.006126 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.006269 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.006249 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.006316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.006297 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64524fe5-c084-40bd-9195-7ca010e99c99-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.007999 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.007983 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.008218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.008201 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64524fe5-c084-40bd-9195-7ca010e99c99-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.067942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.067868 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.068170 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.068142 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82m9\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-kube-api-access-l82m9\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.104063 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.104020 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:18.526769 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:44:18.526732 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64524fe5_c084_40bd_9195_7ca010e99c99.slice/crio-4ac030e3aba92f36e66d211033ef02d3f0f238ddaf947facb5c10e2bb756b795 WatchSource:0}: Error finding container 4ac030e3aba92f36e66d211033ef02d3f0f238ddaf947facb5c10e2bb756b795: Status 404 returned error can't find the container with id 4ac030e3aba92f36e66d211033ef02d3f0f238ddaf947facb5c10e2bb756b795 Nov 02 19:44:18.529159 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.529135 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb"] Nov 02 19:44:18.529439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.529402 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:44:18.529535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.529514 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:44:18.529579 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.529566 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:44:18.616369 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.616333 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" event={"ID":"64524fe5-c084-40bd-9195-7ca010e99c99","Type":"ContainerStarted","Data":"0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1"} Nov 02 19:44:18.616589 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.616420 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" event={"ID":"64524fe5-c084-40bd-9195-7ca010e99c99","Type":"ContainerStarted","Data":"4ac030e3aba92f36e66d211033ef02d3f0f238ddaf947facb5c10e2bb756b795"} Nov 02 19:44:18.735721 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:18.735658 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podStartSLOduration=1.735641529 podStartE2EDuration="1.735641529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:44:18.735245668 +0000 UTC m=+917.077634750" watchObservedRunningTime="2026-11-02 19:44:18.735641529 +0000 UTC m=+917.078030611" Nov 02 19:44:19.104696 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:19.104657 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:19.106283 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:19.106254 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:44:19.106409 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:19.106320 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 19:44:20.104870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:20.104832 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:44:20.105284 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:20.104894 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 19:44:21.104591 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:21.104553 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:44:21.104773 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:21.104618 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 19:44:21.731535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:21.730378 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:22.105212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:22.105131 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:44:22.105212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:22.105184 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 19:44:23.104756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:23.104719 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:44:23.105176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:23.104788 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 19:44:24.104415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:24.104376 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:44:24.104656 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:24.104437 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 19:44:25.105350 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:25.105314 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:44:25.105751 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:25.105370 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 19:44:26.108780 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:26.108742 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:26.109190 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:26.109129 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:26.109895 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:26.109875 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 19:44:26.425000 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:26.424949 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:30.844841 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:30.844804 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:35.293448 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:35.293410 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:37.548971 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.548934 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf"] Nov 02 19:44:37.552656 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.552639 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.609980 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.609942 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"openshift-service-ca.crt\"" Nov 02 19:44:37.654294 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.654254 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-model-cache\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.654522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.654376 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-home\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.654522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.654418 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc565ff-fda0-4743-add0-891392bc36dc-tls-certs\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.654522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.654453 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9lx\" (UniqueName: \"kubernetes.io/projected/2dc565ff-fda0-4743-add0-891392bc36dc-kube-api-access-gv9lx\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.654646 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.654592 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-tmp-dir\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.654681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.654654 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-dshm\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.656197 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.656177 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:37.661663 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.661643 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"kube-root-ca.crt\"" Nov 02 19:44:37.661760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.661681 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"default-dockercfg-824k2\"" Nov 02 19:44:37.661760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.661706 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"distinct-w0-43ceaa-kserve-self-signed-certs\"" Nov 02 19:44:37.663961 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.663941 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf"] Nov 02 19:44:37.755298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.755257 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-model-cache\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.755545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.755321 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-home\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.755545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.755349 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc565ff-fda0-4743-add0-891392bc36dc-tls-certs\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.755545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.755373 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9lx\" (UniqueName: \"kubernetes.io/projected/2dc565ff-fda0-4743-add0-891392bc36dc-kube-api-access-gv9lx\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.755545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.755399 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-tmp-dir\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.755545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.755429 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-dshm\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.755836 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.755791 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-model-cache\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.755882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.755801 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-home\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.755920 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.755900 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-tmp-dir\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.757790 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.757763 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-dshm\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.757957 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.757937 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc565ff-fda0-4743-add0-891392bc36dc-tls-certs\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.813967 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.813888 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9lx\" (UniqueName: \"kubernetes.io/projected/2dc565ff-fda0-4743-add0-891392bc36dc-kube-api-access-gv9lx\") pod \"distinct-w0-43ceaa-kserve-b7b75d898-nzjqf\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:37.866046 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:37.866012 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:38.290833 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:44:38.290790 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dc565ff_fda0_4743_add0_891392bc36dc.slice/crio-932f1b674a4877ebb0200823b0fac62ad11f329fef659af104ca250b625c5c3a WatchSource:0}: Error finding container 932f1b674a4877ebb0200823b0fac62ad11f329fef659af104ca250b625c5c3a: Status 404 returned error can't find the container with id 932f1b674a4877ebb0200823b0fac62ad11f329fef659af104ca250b625c5c3a Nov 02 19:44:38.291957 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:38.291929 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf"] Nov 02 19:44:38.746821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:38.746788 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" event={"ID":"2dc565ff-fda0-4743-add0-891392bc36dc","Type":"ContainerStarted","Data":"ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00"} Nov 02 19:44:38.746821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:38.746828 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" event={"ID":"2dc565ff-fda0-4743-add0-891392bc36dc","Type":"ContainerStarted","Data":"932f1b674a4877ebb0200823b0fac62ad11f329fef659af104ca250b625c5c3a"} Nov 02 19:44:38.865647 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:38.865569 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podStartSLOduration=1.865553867 podStartE2EDuration="1.865553867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:44:38.863649672 +0000 UTC m=+937.206038731" watchObservedRunningTime="2026-11-02 19:44:38.865553867 +0000 UTC m=+937.207942946" Nov 02 19:44:38.866656 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:38.866631 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:39.731290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:39.731253 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:42.086762 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:42.086727 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:42.246845 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:42.246797 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:44.146312 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:44.146269 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:44.395915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:44.395872 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:44.762546 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:44:44.762503 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:44:47.866500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:47.866446 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:47.879514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:47.879469 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:48.638180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:48.638127 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:48.814265 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:48.814235 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:44:49.722411 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:49.722378 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:53.038935 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:53.038885 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:44:57.429208 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:44:57.429172 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:00.246340 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.246300 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr"] Nov 02 19:45:00.250428 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.250408 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:00.300802 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.300755 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm"] Nov 02 19:45:00.304435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.304414 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:00.340626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.340591 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr"] Nov 02 19:45:00.354498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.354440 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0726b6-ce12-4861-88c8-9635a391fdb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr\" (UID: \"8b0726b6-ce12-4861-88c8-9635a391fdb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:00.354683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.354657 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgfz\" (UniqueName: \"kubernetes.io/projected/8b0726b6-ce12-4861-88c8-9635a391fdb0-kube-api-access-8qgfz\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr\" (UID: \"8b0726b6-ce12-4861-88c8-9635a391fdb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:00.354683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.354680 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqnf\" (UniqueName: \"kubernetes.io/projected/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-kube-api-access-vzqnf\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm\" (UID: \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:00.354799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.354704 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm\" (UID: \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:00.374333 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.374290 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm"] Nov 02 19:45:00.383764 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.383731 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:45:00.455331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.455298 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgfz\" (UniqueName: \"kubernetes.io/projected/8b0726b6-ce12-4861-88c8-9635a391fdb0-kube-api-access-8qgfz\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr\" (UID: \"8b0726b6-ce12-4861-88c8-9635a391fdb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:00.455331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.455334 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqnf\" (UniqueName: \"kubernetes.io/projected/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-kube-api-access-vzqnf\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm\" (UID: \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:00.455556 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.455356 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm\" (UID: \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:00.455556 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.455403 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0726b6-ce12-4861-88c8-9635a391fdb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr\" (UID: \"8b0726b6-ce12-4861-88c8-9635a391fdb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:00.456048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.456028 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm\" (UID: \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:00.456091 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.456047 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0726b6-ce12-4861-88c8-9635a391fdb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr\" (UID: \"8b0726b6-ce12-4861-88c8-9635a391fdb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:00.514690 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.514610 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqnf\" (UniqueName: \"kubernetes.io/projected/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-kube-api-access-vzqnf\") pod \"maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm\" (UID: \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:00.514835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.514717 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgfz\" (UniqueName: \"kubernetes.io/projected/8b0726b6-ce12-4861-88c8-9635a391fdb0-kube-api-access-8qgfz\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr\" (UID: \"8b0726b6-ce12-4861-88c8-9635a391fdb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:00.565268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.565229 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:00.619126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.618403 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:00.936156 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:45:00.936122 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0726b6_ce12_4861_88c8_9635a391fdb0.slice/crio-6d3ee490c8e0e9fd42863eea0bc255b1696944c7e5e94b503885cbb5825ade18 WatchSource:0}: Error finding container 6d3ee490c8e0e9fd42863eea0bc255b1696944c7e5e94b503885cbb5825ade18: Status 404 returned error can't find the container with id 6d3ee490c8e0e9fd42863eea0bc255b1696944c7e5e94b503885cbb5825ade18 Nov 02 19:45:00.937778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.937744 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr"] Nov 02 19:45:00.938118 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:00.938101 5850 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:45:01.042240 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:45:01.042208 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode963ecfd_9bfe_4c0f_a4a5_0e75353409e4.slice/crio-264bc43c46c73af92d31727965b3663d8f4bbcc2ec2dfef9400c98db86d32ee5 WatchSource:0}: Error finding container 264bc43c46c73af92d31727965b3663d8f4bbcc2ec2dfef9400c98db86d32ee5: Status 404 returned error can't find the container with id 264bc43c46c73af92d31727965b3663d8f4bbcc2ec2dfef9400c98db86d32ee5 Nov 02 19:45:01.042353 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:01.042270 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm"] Nov 02 19:45:01.832915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:01.832883 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:01.890759 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:01.890723 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" event={"ID":"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4","Type":"ContainerStarted","Data":"264bc43c46c73af92d31727965b3663d8f4bbcc2ec2dfef9400c98db86d32ee5"} Nov 02 19:45:01.892062 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:01.892035 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" event={"ID":"8b0726b6-ce12-4861-88c8-9635a391fdb0","Type":"ContainerStarted","Data":"6d3ee490c8e0e9fd42863eea0bc255b1696944c7e5e94b503885cbb5825ade18"} Nov 02 19:45:02.892551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:02.892501 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:03.048189 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:03.048154 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:03.977098 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:03.977062 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94"] Nov 02 19:45:03.981163 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:03.981143 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.087733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.087700 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1d869e\"/\"distinct-w5-1d869e-kserve-self-signed-certs\"" Nov 02 19:45:04.089091 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.089067 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:04.090224 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.090204 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94"] Nov 02 19:45:04.107780 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.107738 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tmp-dir\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.107943 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.107840 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-model-cache\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.107994 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.107943 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tls-certs\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.107994 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.107968 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-dshm\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.108102 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.107991 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqnmn\" (UniqueName: \"kubernetes.io/projected/f2079ca4-fcae-4d5e-8091-78fddd27fe84-kube-api-access-mqnmn\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.108169 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.108144 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-home\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.209368 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.209326 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tls-certs\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.209368 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.209367 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-dshm\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.209631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.209390 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqnmn\" (UniqueName: \"kubernetes.io/projected/f2079ca4-fcae-4d5e-8091-78fddd27fe84-kube-api-access-mqnmn\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.209631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.209412 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-home\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.209631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.209523 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tmp-dir\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.209631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.209603 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-model-cache\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.209964 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.209935 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-home\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.209964 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.209955 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tmp-dir\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.210055 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.209978 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-model-cache\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.212248 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.212221 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-dshm\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.212565 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.212548 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tls-certs\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.269371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.269272 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqnmn\" (UniqueName: \"kubernetes.io/projected/f2079ca4-fcae-4d5e-8091-78fddd27fe84-kube-api-access-mqnmn\") pod \"distinct-w5-1d869e-kserve-d7788465-6fw94\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.292954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.292912 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:04.760497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.742383 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94"] Nov 02 19:45:04.773544 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:45:04.773493 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2079ca4_fcae_4d5e_8091_78fddd27fe84.slice/crio-19dc4e86aa2ffd3fdd8e51b40a3e8dec68f90d4fcf4bd409d5e7a6e680e4aeaa WatchSource:0}: Error finding container 19dc4e86aa2ffd3fdd8e51b40a3e8dec68f90d4fcf4bd409d5e7a6e680e4aeaa: Status 404 returned error can't find the container with id 19dc4e86aa2ffd3fdd8e51b40a3e8dec68f90d4fcf4bd409d5e7a6e680e4aeaa Nov 02 19:45:04.915374 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.915319 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" event={"ID":"f2079ca4-fcae-4d5e-8091-78fddd27fe84","Type":"ContainerStarted","Data":"3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a"} Nov 02 19:45:04.915374 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:04.915363 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" event={"ID":"f2079ca4-fcae-4d5e-8091-78fddd27fe84","Type":"ContainerStarted","Data":"19dc4e86aa2ffd3fdd8e51b40a3e8dec68f90d4fcf4bd409d5e7a6e680e4aeaa"} Nov 02 19:45:05.040660 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:05.038848 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podStartSLOduration=2.038827204 podStartE2EDuration="2.038827204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:05.038636053 +0000 UTC m=+963.381025232" watchObservedRunningTime="2026-11-02 19:45:05.038827204 +0000 UTC m=+963.381216286" Nov 02 19:45:05.297492 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:05.297299 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:06.490367 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:06.490326 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:07.943147 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:07.943106 5850 generic.go:356] "Generic (PLEG): container finished" podID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerID="971adcb875150e9ad70e169eca98f893556c169d46a09d82691686a4c4448857" exitCode=60 Nov 02 19:45:07.943623 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:07.943179 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" event={"ID":"8b0726b6-ce12-4861-88c8-9635a391fdb0","Type":"ContainerDied","Data":"971adcb875150e9ad70e169eca98f893556c169d46a09d82691686a4c4448857"} Nov 02 19:45:07.943623 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:07.943518 5850 scope.go:122] "RemoveContainer" containerID="971adcb875150e9ad70e169eca98f893556c169d46a09d82691686a4c4448857" Nov 02 19:45:07.945140 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:07.945113 5850 generic.go:356] "Generic (PLEG): container finished" podID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerID="e4452d04c14b091af0737de157eeacb8bdca67821ff59032001a7fec196f35aa" exitCode=60 Nov 02 19:45:07.945268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:07.945167 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" event={"ID":"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4","Type":"ContainerDied","Data":"e4452d04c14b091af0737de157eeacb8bdca67821ff59032001a7fec196f35aa"} Nov 02 19:45:07.945604 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:07.945582 5850 scope.go:122] "RemoveContainer" containerID="e4452d04c14b091af0737de157eeacb8bdca67821ff59032001a7fec196f35aa" Nov 02 19:45:08.622810 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.622778 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:08.791695 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.791660 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:08.954034 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.953937 5850 generic.go:356] "Generic (PLEG): container finished" podID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerID="ab680c3e8f7c6a5b3a2c3aaa1e331dc5b542dcc2f71633243685cd2d90db3603" exitCode=60 Nov 02 19:45:08.954530 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.954027 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" event={"ID":"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4","Type":"ContainerDied","Data":"ab680c3e8f7c6a5b3a2c3aaa1e331dc5b542dcc2f71633243685cd2d90db3603"} Nov 02 19:45:08.954530 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.954094 5850 scope.go:122] "RemoveContainer" containerID="e4452d04c14b091af0737de157eeacb8bdca67821ff59032001a7fec196f35aa" Nov 02 19:45:08.954530 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.954391 5850 scope.go:122] "RemoveContainer" containerID="ab680c3e8f7c6a5b3a2c3aaa1e331dc5b542dcc2f71633243685cd2d90db3603" Nov 02 19:45:08.954789 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:45:08.954767 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm_odh-ai-gateway-infra(e963ecfd-9bfe-4c0f-a4a5-0e75353409e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" Nov 02 19:45:08.956855 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.956833 5850 generic.go:356] "Generic (PLEG): container finished" podID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerID="e1d7b89c4e8bf55f461cae387eaa5ed6458a09610456a89e235992d2ea13697d" exitCode=60 Nov 02 19:45:08.956958 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.956915 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" event={"ID":"8b0726b6-ce12-4861-88c8-9635a391fdb0","Type":"ContainerDied","Data":"e1d7b89c4e8bf55f461cae387eaa5ed6458a09610456a89e235992d2ea13697d"} Nov 02 19:45:08.957189 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.957176 5850 scope.go:122] "RemoveContainer" containerID="e1d7b89c4e8bf55f461cae387eaa5ed6458a09610456a89e235992d2ea13697d" Nov 02 19:45:08.957430 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:45:08.957413 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr_odh-ai-gateway-infra(8b0726b6-ce12-4861-88c8-9635a391fdb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" Nov 02 19:45:08.967027 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:08.967004 5850 scope.go:122] "RemoveContainer" containerID="971adcb875150e9ad70e169eca98f893556c169d46a09d82691686a4c4448857" Nov 02 19:45:09.963890 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:09.963860 5850 scope.go:122] "RemoveContainer" containerID="ab680c3e8f7c6a5b3a2c3aaa1e331dc5b542dcc2f71633243685cd2d90db3603" Nov 02 19:45:09.964363 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:45:09.964174 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm_odh-ai-gateway-infra(e963ecfd-9bfe-4c0f-a4a5-0e75353409e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" Nov 02 19:45:09.965415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:09.965391 5850 scope.go:122] "RemoveContainer" containerID="e1d7b89c4e8bf55f461cae387eaa5ed6458a09610456a89e235992d2ea13697d" Nov 02 19:45:09.965681 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:45:09.965660 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr_odh-ai-gateway-infra(8b0726b6-ce12-4861-88c8-9635a391fdb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" Nov 02 19:45:11.036776 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:11.036744 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:14.293224 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:14.293186 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:14.306087 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:14.306058 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:14.761059 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:45:14.761020 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:45:15.004336 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:15.004307 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:45:15.450411 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:15.450379 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:17.718483 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.718449 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8"] Nov 02 19:45:17.725410 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.724015 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.779090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.779059 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8"] Nov 02 19:45:17.808609 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.808572 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tls-certs\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.808790 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.808622 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jfv\" (UniqueName: \"kubernetes.io/projected/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-kube-api-access-g9jfv\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.808839 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.808791 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tmp-dir\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.808887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.808846 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-home\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.808949 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.808935 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-model-cache\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.808981 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.808969 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-dshm\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.820359 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.820317 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:17.828355 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.828329 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"distinct-w2-28623b-kserve-self-signed-certs\"" Nov 02 19:45:17.909918 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.909881 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tmp-dir\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.910083 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.909949 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-home\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.910083 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.909990 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-model-cache\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.910083 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.910011 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-dshm\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.910252 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.910117 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tls-certs\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.910252 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.910144 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jfv\" (UniqueName: \"kubernetes.io/projected/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-kube-api-access-g9jfv\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.910373 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.910334 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tmp-dir\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.910434 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.910393 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-home\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.910434 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.910418 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-model-cache\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.912577 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.912555 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-dshm\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.912724 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.912704 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tls-certs\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:17.971704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:17.971630 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jfv\" (UniqueName: \"kubernetes.io/projected/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-kube-api-access-g9jfv\") pod \"distinct-w2-28623b-kserve-5f7845b8d4-b7qm8\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:18.046030 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:18.045986 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:18.465355 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:45:18.465320 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1fef2ff_3579_43ff_b49b_d86e0d0698e0.slice/crio-f361ae1b8e6b0edf8e8988d09ebce052621cc25dab0ceb30fd064e0fe3f0f168 WatchSource:0}: Error finding container f361ae1b8e6b0edf8e8988d09ebce052621cc25dab0ceb30fd064e0fe3f0f168: Status 404 returned error can't find the container with id f361ae1b8e6b0edf8e8988d09ebce052621cc25dab0ceb30fd064e0fe3f0f168 Nov 02 19:45:18.467276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:18.467250 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8"] Nov 02 19:45:19.018765 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:19.018728 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" event={"ID":"f1fef2ff-3579-43ff-b49b-d86e0d0698e0","Type":"ContainerStarted","Data":"0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f"} Nov 02 19:45:19.019410 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:19.018785 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" event={"ID":"f1fef2ff-3579-43ff-b49b-d86e0d0698e0","Type":"ContainerStarted","Data":"f361ae1b8e6b0edf8e8988d09ebce052621cc25dab0ceb30fd064e0fe3f0f168"} Nov 02 19:45:19.046714 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:19.046680 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:19.139998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:19.139944 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podStartSLOduration=2.13992866 podStartE2EDuration="2.13992866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:19.137979664 +0000 UTC m=+977.480368744" watchObservedRunningTime="2026-11-02 19:45:19.13992866 +0000 UTC m=+977.482317743" Nov 02 19:45:19.955981 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:19.955941 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:22.808680 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:22.808635 5850 scope.go:122] "RemoveContainer" containerID="e1d7b89c4e8bf55f461cae387eaa5ed6458a09610456a89e235992d2ea13697d" Nov 02 19:45:22.809077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:22.808733 5850 scope.go:122] "RemoveContainer" containerID="ab680c3e8f7c6a5b3a2c3aaa1e331dc5b542dcc2f71633243685cd2d90db3603" Nov 02 19:45:24.051741 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:24.051654 5850 generic.go:356] "Generic (PLEG): container finished" podID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerID="6430179f090f87e206b12812620cfe7390d46139282fc9df436701f9e142d9e3" exitCode=60 Nov 02 19:45:24.051741 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:24.051730 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" event={"ID":"8b0726b6-ce12-4861-88c8-9635a391fdb0","Type":"ContainerDied","Data":"6430179f090f87e206b12812620cfe7390d46139282fc9df436701f9e142d9e3"} Nov 02 19:45:24.052266 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:24.051794 5850 scope.go:122] "RemoveContainer" containerID="e1d7b89c4e8bf55f461cae387eaa5ed6458a09610456a89e235992d2ea13697d" Nov 02 19:45:24.052266 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:24.052220 5850 scope.go:122] "RemoveContainer" containerID="6430179f090f87e206b12812620cfe7390d46139282fc9df436701f9e142d9e3" Nov 02 19:45:24.052575 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:45:24.052546 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr_odh-ai-gateway-infra(8b0726b6-ce12-4861-88c8-9635a391fdb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" Nov 02 19:45:24.053904 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:24.053882 5850 generic.go:356] "Generic (PLEG): container finished" podID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerID="7328fe27fdb4e360a7099f4813f7fe2ea8521eb3bb9b10ed8c011144c0b6c30b" exitCode=60 Nov 02 19:45:24.054000 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:24.053923 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" event={"ID":"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4","Type":"ContainerDied","Data":"7328fe27fdb4e360a7099f4813f7fe2ea8521eb3bb9b10ed8c011144c0b6c30b"} Nov 02 19:45:24.054240 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:24.054227 5850 scope.go:122] "RemoveContainer" containerID="7328fe27fdb4e360a7099f4813f7fe2ea8521eb3bb9b10ed8c011144c0b6c30b" Nov 02 19:45:24.054527 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:45:24.054506 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm_odh-ai-gateway-infra(e963ecfd-9bfe-4c0f-a4a5-0e75353409e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" Nov 02 19:45:24.063987 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:24.063958 5850 scope.go:122] "RemoveContainer" containerID="ab680c3e8f7c6a5b3a2c3aaa1e331dc5b542dcc2f71633243685cd2d90db3603" Nov 02 19:45:24.436113 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:24.436078 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:25.179511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.179461 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr"] Nov 02 19:45:25.296265 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.296232 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm"] Nov 02 19:45:25.332630 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.332604 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:25.401503 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.401430 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0726b6-ce12-4861-88c8-9635a391fdb0-kube-api-access-8qgfz\" (UniqueName: \"kubernetes.io/projected/8b0726b6-ce12-4861-88c8-9635a391fdb0-kube-api-access-8qgfz\") pod \"8b0726b6-ce12-4861-88c8-9635a391fdb0\" (UID: \"8b0726b6-ce12-4861-88c8-9635a391fdb0\") " Nov 02 19:45:25.401695 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.401592 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0726b6-ce12-4861-88c8-9635a391fdb0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0726b6-ce12-4861-88c8-9635a391fdb0-maas-api-service-ca\") pod \"8b0726b6-ce12-4861-88c8-9635a391fdb0\" (UID: \"8b0726b6-ce12-4861-88c8-9635a391fdb0\") " Nov 02 19:45:25.402087 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.402058 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0726b6-ce12-4861-88c8-9635a391fdb0-maas-api-service-ca" pod "8b0726b6-ce12-4861-88c8-9635a391fdb0" (UID: "8b0726b6-ce12-4861-88c8-9635a391fdb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:45:25.403771 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.403747 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0726b6-ce12-4861-88c8-9635a391fdb0-kube-api-access-8qgfz" pod "8b0726b6-ce12-4861-88c8-9635a391fdb0" (UID: "8b0726b6-ce12-4861-88c8-9635a391fdb0"). InnerVolumeSpecName "kube-api-access-8qgfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:45:25.456874 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.456795 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:25.502931 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.502884 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-maas-api-service-ca\") pod \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\" (UID: \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\") " Nov 02 19:45:25.503120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.503000 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-kube-api-access-vzqnf\" (UniqueName: \"kubernetes.io/projected/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-kube-api-access-vzqnf\") pod \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\" (UID: \"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4\") " Nov 02 19:45:25.503341 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.503316 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-maas-api-service-ca" pod "e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" (UID: "e963ecfd-9bfe-4c0f-a4a5-0e75353409e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:45:25.503533 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.503515 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0726b6-ce12-4861-88c8-9635a391fdb0-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:25.503599 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.503538 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:25.503599 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.503553 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgfz\" (UniqueName: \"kubernetes.io/projected/8b0726b6-ce12-4861-88c8-9635a391fdb0-kube-api-access-8qgfz\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:25.505206 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.505185 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-kube-api-access-vzqnf" pod "e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" (UID: "e963ecfd-9bfe-4c0f-a4a5-0e75353409e4"). InnerVolumeSpecName "kube-api-access-vzqnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:45:25.604706 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:25.604671 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzqnf\" (UniqueName: \"kubernetes.io/projected/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4-kube-api-access-vzqnf\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:26.069701 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.069671 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" Nov 02 19:45:26.069701 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.069680 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr" event={"ID":"8b0726b6-ce12-4861-88c8-9635a391fdb0","Type":"ContainerDied","Data":"6d3ee490c8e0e9fd42863eea0bc255b1696944c7e5e94b503885cbb5825ade18"} Nov 02 19:45:26.069938 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.069755 5850 scope.go:122] "RemoveContainer" containerID="6430179f090f87e206b12812620cfe7390d46139282fc9df436701f9e142d9e3" Nov 02 19:45:26.071306 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.071283 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" event={"ID":"e963ecfd-9bfe-4c0f-a4a5-0e75353409e4","Type":"ContainerDied","Data":"264bc43c46c73af92d31727965b3663d8f4bbcc2ec2dfef9400c98db86d32ee5"} Nov 02 19:45:26.071306 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.071300 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm" Nov 02 19:45:26.078876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.078826 5850 scope.go:122] "RemoveContainer" containerID="7328fe27fdb4e360a7099f4813f7fe2ea8521eb3bb9b10ed8c011144c0b6c30b" Nov 02 19:45:26.239537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.239497 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm"] Nov 02 19:45:26.242649 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.242621 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-28623b-29894145-fhqqm"] Nov 02 19:45:26.354913 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.354830 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr"] Nov 02 19:45:26.356826 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:26.356798 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894145-275xr"] Nov 02 19:45:27.815440 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:27.815409 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" path="/var/lib/kubelet/pods/8b0726b6-ce12-4861-88c8-9635a391fdb0/volumes" Nov 02 19:45:27.815991 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:27.815975 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" path="/var/lib/kubelet/pods/e963ecfd-9bfe-4c0f-a4a5-0e75353409e4/volumes" Nov 02 19:45:28.046833 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:28.046799 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:28.060154 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:28.060123 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:28.096516 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:28.096410 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:45:28.937756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:28.937727 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:33.332759 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:33.332726 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:37.738301 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:37.738264 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:38.081576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:38.081488 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:42.137857 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:42.137776 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:44.760973 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:45:44.760935 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:45:46.534116 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:46.534083 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:49.761813 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.761772 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8"] Nov 02 19:45:49.762279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762265 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerName="cleanup" Nov 02 19:45:49.762322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762280 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerName="cleanup" Nov 02 19:45:49.762322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762307 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerName="cleanup" Nov 02 19:45:49.762322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762312 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerName="cleanup" Nov 02 19:45:49.762322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762322 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerName="cleanup" Nov 02 19:45:49.762444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762328 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerName="cleanup" Nov 02 19:45:49.762444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762345 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerName="cleanup" Nov 02 19:45:49.762444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762350 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerName="cleanup" Nov 02 19:45:49.762444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762426 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerName="cleanup" Nov 02 19:45:49.762444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762433 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerName="cleanup" Nov 02 19:45:49.762444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762439 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerName="cleanup" Nov 02 19:45:49.762674 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.762449 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerName="cleanup" Nov 02 19:45:49.765749 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.765720 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.796564 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.796533 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.796819 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.796703 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.796819 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.796735 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.796819 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.796754 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.796819 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.796770 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjlq\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-kube-api-access-sbjlq\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.796955 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.796835 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.796955 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.796865 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.797018 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.796956 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.797018 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.796998 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.876489 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.876443 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8"] Nov 02 19:45:49.897993 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.897940 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.897993 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.897992 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898217 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898086 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898217 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898109 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjlq\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-kube-api-access-sbjlq\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898217 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898145 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898229 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898324 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898365 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898407 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898430 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898456 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898762 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898742 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.898821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.898809 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.899314 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.899287 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.900748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.900727 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.900897 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.900877 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.956418 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.956383 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:49.956606 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:49.956553 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjlq\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-kube-api-access-sbjlq\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:50.081482 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:50.081379 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:50.507243 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:45:50.507208 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ced20f2_def6_42a7_9c08_96345ef8a5a1.slice/crio-945f0ea13406f75059d1bf7f2e8aacba028d326dc5100dd74c0030f03c233ade WatchSource:0}: Error finding container 945f0ea13406f75059d1bf7f2e8aacba028d326dc5100dd74c0030f03c233ade: Status 404 returned error can't find the container with id 945f0ea13406f75059d1bf7f2e8aacba028d326dc5100dd74c0030f03c233ade Nov 02 19:45:50.509028 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:50.509002 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8"] Nov 02 19:45:50.509692 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:50.509659 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:45:50.509761 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:50.509721 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:45:50.509761 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:50.509751 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:45:50.939256 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:50.939226 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:51.266454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:51.266364 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" event={"ID":"0ced20f2-def6-42a7-9c08-96345ef8a5a1","Type":"ContainerStarted","Data":"60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513"} Nov 02 19:45:51.266454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:51.266428 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" event={"ID":"0ced20f2-def6-42a7-9c08-96345ef8a5a1","Type":"ContainerStarted","Data":"945f0ea13406f75059d1bf7f2e8aacba028d326dc5100dd74c0030f03c233ade"} Nov 02 19:45:51.439241 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:51.439182 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" podStartSLOduration=2.439164216 podStartE2EDuration="2.439164216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:51.436336563 +0000 UTC m=+1009.778725642" watchObservedRunningTime="2026-11-02 19:45:51.439164216 +0000 UTC m=+1009.781553297" Nov 02 19:45:52.082289 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:52.082230 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:52.084199 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:52.084173 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 19:45:52.084326 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:52.084219 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 19:45:53.082259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:53.082223 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 19:45:53.082414 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:53.082278 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 19:45:54.082319 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:54.082281 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 19:45:54.082500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:54.082346 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 19:45:55.081874 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:55.081841 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 19:45:55.082065 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:55.081904 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 19:45:55.344188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:55.344066 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:45:56.082936 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:56.082895 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 19:45:56.084107 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:56.084073 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 19:45:57.081760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:57.081712 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 19:45:57.082325 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:57.081771 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 19:45:58.087020 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:58.086991 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:58.087444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:58.087273 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:58.087970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:58.087949 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:45:59.856846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:45:59.856816 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:04.267767 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.267725 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:04.820786 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.820754 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2"] Nov 02 19:46:04.821248 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.821236 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerName="cleanup" Nov 02 19:46:04.821295 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.821250 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerName="cleanup" Nov 02 19:46:04.821330 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.821300 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerName="cleanup" Nov 02 19:46:04.821330 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.821308 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerName="cleanup" Nov 02 19:46:04.821404 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.821394 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0726b6-ce12-4861-88c8-9635a391fdb0" containerName="cleanup" Nov 02 19:46:04.821443 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.821412 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e963ecfd-9bfe-4c0f-a4a5-0e75353409e4" containerName="cleanup" Nov 02 19:46:04.826101 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.826081 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.880132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.880095 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-28623b-openshift-default-dockercfg-cxdg7\"" Nov 02 19:46:04.884742 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.884715 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.884863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.884839 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6a4b92-1091-419e-ba15-1a6af5381698-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.884954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.884935 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.885134 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.885116 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.885171 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.885149 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.885209 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.885169 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.885290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.885276 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.885345 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.885313 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.885427 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.885407 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlkk2\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-kube-api-access-hlkk2\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.933975 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.933939 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2"] Nov 02 19:46:04.986918 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.986881 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987069 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.986989 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987069 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987017 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987069 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987039 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987086 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987115 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987141 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkk2\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-kube-api-access-hlkk2\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987213 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987426 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987291 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6a4b92-1091-419e-ba15-1a6af5381698-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987426 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987341 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987426 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987390 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987605 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.987679 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987627 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.988006 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.987984 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6a4b92-1091-419e-ba15-1a6af5381698-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.989590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.989561 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:04.989782 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:04.989761 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:05.045390 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:05.045355 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:05.045569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:05.045424 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlkk2\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-kube-api-access-hlkk2\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:05.140110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:05.140070 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:05.560880 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:46:05.560846 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6a4b92_1091_419e_ba15_1a6af5381698.slice/crio-612ccb6fc10c2969394e203edea87e2d4fa266a164484dc5ee55e22bea53aa33 WatchSource:0}: Error finding container 612ccb6fc10c2969394e203edea87e2d4fa266a164484dc5ee55e22bea53aa33: Status 404 returned error can't find the container with id 612ccb6fc10c2969394e203edea87e2d4fa266a164484dc5ee55e22bea53aa33 Nov 02 19:46:05.563176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:05.563152 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2"] Nov 02 19:46:05.563792 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:05.563758 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:46:05.563845 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:05.563825 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:46:05.563882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:05.563855 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:46:06.359504 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:06.359446 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" event={"ID":"dd6a4b92-1091-419e-ba15-1a6af5381698","Type":"ContainerStarted","Data":"dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e"} Nov 02 19:46:06.359713 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:06.359516 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" event={"ID":"dd6a4b92-1091-419e-ba15-1a6af5381698","Type":"ContainerStarted","Data":"612ccb6fc10c2969394e203edea87e2d4fa266a164484dc5ee55e22bea53aa33"} Nov 02 19:46:06.584809 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:06.584753 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" podStartSLOduration=2.5847376029999998 podStartE2EDuration="2.584737603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:46:06.532048086 +0000 UTC m=+1024.874437192" watchObservedRunningTime="2026-11-02 19:46:06.584737603 +0000 UTC m=+1024.927126683" Nov 02 19:46:07.140970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:07.140931 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:07.143024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:07.142990 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 19:46:07.143164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:07.143038 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 19:46:08.140765 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:08.140728 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 19:46:08.141170 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:08.140784 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 19:46:08.732896 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:08.732821 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:09.039913 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:09.039824 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:09.140509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:09.140456 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 19:46:09.140682 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:09.140535 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 19:46:09.431501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:09.430405 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:10.141331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:10.141285 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 19:46:10.141569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:10.141358 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 19:46:11.140576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:11.140542 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 19:46:11.140996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:11.140599 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 19:46:12.140932 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:12.140894 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 19:46:12.141412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:12.140972 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 19:46:13.145066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:13.145024 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:13.145445 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:13.145304 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:13.145952 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:13.145937 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:46:13.157284 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:13.157248 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:14.762255 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:46:14.762222 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:46:17.565831 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:17.565797 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:21.468052 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:21.468013 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:22.063911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:22.063868 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:26.592352 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:26.592315 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:31.034782 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:31.034733 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:35.689658 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:35.689617 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:40.035016 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:40.034981 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:44.443927 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:44.443886 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:44.760832 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:46:44.760742 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:46:49.056047 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:49.056014 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:53.620580 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:53.620540 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:54.327237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:54.327201 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:46:58.051489 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:46:58.051446 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:02.454449 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:02.454415 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:04.019331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.019292 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw"] Nov 02 19:47:04.024937 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.024912 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.139465 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.138950 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw"] Nov 02 19:47:04.161828 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.161767 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.161828 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.161823 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.162090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.161921 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5436015e-f85e-437c-b88c-7a30ddc02c77-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.162090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.161979 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7clpn\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-kube-api-access-7clpn\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.162090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.162056 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.162261 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.162115 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.162261 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.162145 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.162261 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.162232 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.162261 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.162259 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.263911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.263876 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5436015e-f85e-437c-b88c-7a30ddc02c77-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264083 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.263936 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7clpn\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-kube-api-access-7clpn\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264083 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.263980 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264162 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264114 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264161 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264205 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264312 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264225 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264312 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264279 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264420 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264316 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264420 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264391 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264604 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264579 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264680 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264661 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264737 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264722 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.264835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.264804 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5436015e-f85e-437c-b88c-7a30ddc02c77-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.266454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.266434 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.266719 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.266699 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.322652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.322581 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.322799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.322690 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7clpn\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-kube-api-access-7clpn\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.338958 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.338931 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:04.766765 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:47:04.766728 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5436015e_f85e_437c_b88c_7a30ddc02c77.slice/crio-b4d163aefb1c04b2560153307f1604684a5f87a4b5e9855dd832d6273acb00af WatchSource:0}: Error finding container b4d163aefb1c04b2560153307f1604684a5f87a4b5e9855dd832d6273acb00af: Status 404 returned error can't find the container with id b4d163aefb1c04b2560153307f1604684a5f87a4b5e9855dd832d6273acb00af Nov 02 19:47:04.769180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.769153 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw"] Nov 02 19:47:04.769815 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.769780 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:04.769885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.769852 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:04.769885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.769881 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:04.936932 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.936897 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6"] Nov 02 19:47:04.941713 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:04.941693 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.041024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.040938 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6"] Nov 02 19:47:05.074163 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.074118 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.074430 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.074409 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.074653 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.074625 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.074821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.074797 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.074887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.074862 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wdqv\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-kube-api-access-8wdqv\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.075024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.074998 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.075080 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.075044 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.075137 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.075089 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.075186 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.075151 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.176362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.176322 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.176574 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.176386 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wdqv\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-kube-api-access-8wdqv\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.176574 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.176507 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.176574 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.176540 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.176574 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.176567 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.176808 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.176598 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.176914 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.176865 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.176974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.176920 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.177026 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.176992 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.177078 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.177039 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.177192 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.177168 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.177275 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.177242 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.177371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.177349 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.177898 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.177871 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.178864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.178844 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.179341 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.179316 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.236300 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.236268 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wdqv\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-kube-api-access-8wdqv\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.236533 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.236269 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.256706 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.256679 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:05.684264 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:47:05.684225 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268a0dec_6781_41b8_b2c9_fe08f8d18ded.slice/crio-63469727a28e4d89710bd66c787ae63dec62ab8750652cb0b8363e0575ca2f98 WatchSource:0}: Error finding container 63469727a28e4d89710bd66c787ae63dec62ab8750652cb0b8363e0575ca2f98: Status 404 returned error can't find the container with id 63469727a28e4d89710bd66c787ae63dec62ab8750652cb0b8363e0575ca2f98 Nov 02 19:47:05.686770 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.686712 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:05.686852 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.686782 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:05.686852 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.686813 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:05.687204 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.687173 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6"] Nov 02 19:47:05.756538 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.756505 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" event={"ID":"268a0dec-6781-41b8-b2c9-fe08f8d18ded","Type":"ContainerStarted","Data":"63469727a28e4d89710bd66c787ae63dec62ab8750652cb0b8363e0575ca2f98"} Nov 02 19:47:05.758258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.758233 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" event={"ID":"5436015e-f85e-437c-b88c-7a30ddc02c77","Type":"ContainerStarted","Data":"f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a"} Nov 02 19:47:05.758382 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.758279 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" event={"ID":"5436015e-f85e-437c-b88c-7a30ddc02c77","Type":"ContainerStarted","Data":"b4d163aefb1c04b2560153307f1604684a5f87a4b5e9855dd832d6273acb00af"} Nov 02 19:47:05.885267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:05.885209 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" podStartSLOduration=2.885192823 podStartE2EDuration="2.885192823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:05.882850807 +0000 UTC m=+1084.225239889" watchObservedRunningTime="2026-11-02 19:47:05.885192823 +0000 UTC m=+1084.227581902" Nov 02 19:47:06.339553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:06.339512 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:06.340910 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:06.340881 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 19:47:06.341050 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:06.340931 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 19:47:06.765966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:06.765923 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" event={"ID":"268a0dec-6781-41b8-b2c9-fe08f8d18ded","Type":"ContainerStarted","Data":"408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f"} Nov 02 19:47:06.867077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:06.867041 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:06.957356 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:06.957272 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" podStartSLOduration=2.9572408770000003 podStartE2EDuration="2.957240877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:06.923779061 +0000 UTC m=+1085.266168165" watchObservedRunningTime="2026-11-02 19:47:06.957240877 +0000 UTC m=+1085.299629958" Nov 02 19:47:07.257079 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:07.257025 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:07.259096 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:07.259067 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 19:47:07.259240 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:07.259129 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 19:47:07.345504 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:07.342379 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 19:47:07.345504 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:07.342461 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 19:47:08.257995 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:08.257956 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 19:47:08.258347 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:08.258314 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 19:47:08.340969 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:08.340926 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 19:47:08.341215 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:08.341187 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 19:47:09.079924 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:09.079888 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:09.257961 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:09.257918 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 19:47:09.258809 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:09.258330 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 19:47:09.342992 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:09.342867 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 19:47:09.343365 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:09.343337 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 19:47:09.424250 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:09.424204 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:10.257256 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:10.257213 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 19:47:10.257939 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:10.257889 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 19:47:10.339926 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:10.339882 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 19:47:10.340207 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:10.340174 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 19:47:11.258028 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:11.257987 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 19:47:11.258650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:11.258043 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 19:47:11.346498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:11.346317 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 19:47:11.346498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:11.346379 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 19:47:11.602680 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:11.602586 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:12.257978 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:12.257930 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 19:47:12.258169 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:12.258008 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 19:47:12.347603 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:12.347563 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 19:47:12.348384 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:12.347883 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 19:47:13.349819 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:13.349783 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:13.351353 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:13.351324 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:13.352873 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:13.352848 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:47:14.258054 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:14.258003 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:47:14.258283 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:14.258059 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:47:14.764013 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:47:14.763962 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:47:14.959168 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:14.959138 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:14.959634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:14.959604 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:14.960771 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:14.960746 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 19:47:16.130975 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:16.130913 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:16.601567 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:16.601457 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:20.548352 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:20.548316 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:21.882501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.882439 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6"] Nov 02 19:47:21.886116 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.886097 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:21.930484 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.930437 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-model-cache\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:21.930664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.930579 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tmp-dir\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:21.930724 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.930658 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-home\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:21.930724 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.930686 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvm5\" (UniqueName: \"kubernetes.io/projected/8ef32e82-ae3e-4e7d-b263-68e656cf1784-kube-api-access-kpvm5\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:21.930805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.930788 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tls-certs\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:21.930863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.930851 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-dshm\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:21.992283 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.992238 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"openshift-service-ca.crt\"" Nov 02 19:47:21.992561 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.992536 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"kube-root-ca.crt\"" Nov 02 19:47:21.992650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.992554 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"default-dockercfg-f89sw\"" Nov 02 19:47:21.992650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.992542 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"distinct-2-w3-5c46af-kserve-self-signed-certs\"" Nov 02 19:47:21.994753 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:21.994733 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6"] Nov 02 19:47:22.016581 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.016544 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:22.058552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.048877 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tmp-dir\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.058898 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.058870 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-home\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.059378 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.059357 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvm5\" (UniqueName: \"kubernetes.io/projected/8ef32e82-ae3e-4e7d-b263-68e656cf1784-kube-api-access-kpvm5\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.059952 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.059929 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tls-certs\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.061111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.055086 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tmp-dir\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.062338 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.059306 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-home\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.062737 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.062705 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-dshm\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.062901 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.062883 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-model-cache\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.063600 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.063551 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-model-cache\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.073013 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.072762 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-dshm\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.075268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.075230 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tls-certs\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.118639 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.118605 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvm5\" (UniqueName: \"kubernetes.io/projected/8ef32e82-ae3e-4e7d-b263-68e656cf1784-kube-api-access-kpvm5\") pod \"distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.198088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.197997 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:22.655258 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:47:22.655226 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ef32e82_ae3e_4e7d_b263_68e656cf1784.slice/crio-0442c8336c63a18e4417a2f8b1030d9f29ab66ff6632a45d70d9b1d5294a20b0 WatchSource:0}: Error finding container 0442c8336c63a18e4417a2f8b1030d9f29ab66ff6632a45d70d9b1d5294a20b0: Status 404 returned error can't find the container with id 0442c8336c63a18e4417a2f8b1030d9f29ab66ff6632a45d70d9b1d5294a20b0 Nov 02 19:47:22.656465 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.656440 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6"] Nov 02 19:47:22.938686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.938587 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" event={"ID":"8ef32e82-ae3e-4e7d-b263-68e656cf1784","Type":"ContainerStarted","Data":"11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4"} Nov 02 19:47:22.938686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:22.938640 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" event={"ID":"8ef32e82-ae3e-4e7d-b263-68e656cf1784","Type":"ContainerStarted","Data":"0442c8336c63a18e4417a2f8b1030d9f29ab66ff6632a45d70d9b1d5294a20b0"} Nov 02 19:47:23.057141 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:23.057077 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podStartSLOduration=2.057063204 podStartE2EDuration="2.057063204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:23.055999152 +0000 UTC m=+1101.398388236" watchObservedRunningTime="2026-11-02 19:47:23.057063204 +0000 UTC m=+1101.399452284" Nov 02 19:47:23.212551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:23.212383 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:24.949709 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:24.949668 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:29.451758 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:29.451719 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:30.728306 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:30.728189 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:32.199162 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:32.199126 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:32.211545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:32.211514 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:33.030415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:33.030384 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:47:33.946883 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:33.946845 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:34.584940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.584907 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx"] Nov 02 19:47:34.589619 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.589600 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.636585 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.636550 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr"] Nov 02 19:47:34.640861 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.640840 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.665678 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.665644 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.665836 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.665685 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.665836 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.665819 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.665933 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.665854 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.665933 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.665872 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.666170 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.666139 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.666279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.666257 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.666348 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.666305 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.666348 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.666327 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtktr\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-kube-api-access-dtktr\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.691357 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.691321 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx"] Nov 02 19:47:34.744086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.744051 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr"] Nov 02 19:47:34.767854 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.767801 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768065 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.767863 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.768065 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.767958 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768065 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768001 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768065 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768051 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.768257 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768080 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.768257 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768119 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.768257 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768244 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768363 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768259 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768406 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768371 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768406 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768400 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768416 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtktr\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-kube-api-access-dtktr\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768458 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.768544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768534 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vqb\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-kube-api-access-r2vqb\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.768707 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768579 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768707 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768628 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768811 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768705 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768811 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768785 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768817 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.768911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768847 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.768911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768867 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.768911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.768876 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.769148 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.769124 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.770717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.770692 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.771258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.771240 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.833362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.833322 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.845241 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.845174 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtktr\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-kube-api-access-dtktr\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.845241 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.845200 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st"] Nov 02 19:47:34.851640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.851616 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:34.869908 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.869881 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.869908 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.869912 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.869936 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870053 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870091 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vqb\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-kube-api-access-r2vqb\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870333 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870313 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870421 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870351 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870639 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870600 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870723 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870634 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870723 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870662 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870838 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870718 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870838 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870816 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.870947 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870867 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.871004 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.870957 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.872539 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.872519 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.872638 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.872564 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.904043 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.904008 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:34.908059 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.907915 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st"] Nov 02 19:47:34.933874 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.929198 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.933874 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.929493 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vqb\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-kube-api-access-r2vqb\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.954002 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.953726 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:34.971829 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.971640 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:34.971829 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.971713 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:34.971829 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.971816 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:34.972012 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.971861 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2k8x\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-kube-api-access-l2k8x\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:34.972012 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.971900 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:34.972012 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.971923 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:34.972012 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.971961 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:34.972135 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.972017 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:34.972135 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:34.972070 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.073267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073236 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.073412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073290 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k8x\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-kube-api-access-l2k8x\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.073412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073315 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.073412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073332 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.073412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073371 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.073654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073427 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.073654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073500 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.073654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073591 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.073801 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073652 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.074126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.073893 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.074126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.074031 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.074126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.074091 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.074352 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.074313 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.074433 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.074410 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.075461 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.075402 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.076497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.076448 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.132899 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.132864 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.132899 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.132895 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2k8x\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-kube-api-access-l2k8x\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.170194 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.170162 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:35.338001 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:47:35.337966 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfd26c4_7283_49e5_bbd6_d69ff19bce71.slice/crio-36375854f3e1d26edf4089c65b39372555bcea10a1242e0f721dbd36fc681039 WatchSource:0}: Error finding container 36375854f3e1d26edf4089c65b39372555bcea10a1242e0f721dbd36fc681039: Status 404 returned error can't find the container with id 36375854f3e1d26edf4089c65b39372555bcea10a1242e0f721dbd36fc681039 Nov 02 19:47:35.338220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.338196 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx"] Nov 02 19:47:35.340512 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.340456 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:35.340581 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.340551 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:35.340613 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.340585 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:35.389720 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:47:35.389686 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43bb82d_4270_4943_ad7f_c16cb35f86fc.slice/crio-48d34ce1c4f975e22f6e86658c9b4b6ad2a75acfdf5f953890dc526a74b0ac2c WatchSource:0}: Error finding container 48d34ce1c4f975e22f6e86658c9b4b6ad2a75acfdf5f953890dc526a74b0ac2c: Status 404 returned error can't find the container with id 48d34ce1c4f975e22f6e86658c9b4b6ad2a75acfdf5f953890dc526a74b0ac2c Nov 02 19:47:35.394211 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.393357 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr"] Nov 02 19:47:35.402188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.402137 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:35.402318 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.402227 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:35.402318 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.402267 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:35.551137 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.550909 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st"] Nov 02 19:47:35.551317 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:47:35.551297 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58af3e1a_198c_4706_9bfa_e5b1ba476d4c.slice/crio-59b02093da115b8bd1f152b888d52def3a15b795b9f729c04d6ca99230522351 WatchSource:0}: Error finding container 59b02093da115b8bd1f152b888d52def3a15b795b9f729c04d6ca99230522351: Status 404 returned error can't find the container with id 59b02093da115b8bd1f152b888d52def3a15b795b9f729c04d6ca99230522351 Nov 02 19:47:35.553349 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.553314 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:35.553432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.553382 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:35.553432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:35.553413 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:47:36.058891 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.058856 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" event={"ID":"d43bb82d-4270-4943-ad7f-c16cb35f86fc","Type":"ContainerStarted","Data":"ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954"} Nov 02 19:47:36.059366 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.058909 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" event={"ID":"d43bb82d-4270-4943-ad7f-c16cb35f86fc","Type":"ContainerStarted","Data":"48d34ce1c4f975e22f6e86658c9b4b6ad2a75acfdf5f953890dc526a74b0ac2c"} Nov 02 19:47:36.060522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.060491 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" event={"ID":"1bfd26c4-7283-49e5-bbd6-d69ff19bce71","Type":"ContainerStarted","Data":"9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2"} Nov 02 19:47:36.060640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.060549 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" event={"ID":"1bfd26c4-7283-49e5-bbd6-d69ff19bce71","Type":"ContainerStarted","Data":"36375854f3e1d26edf4089c65b39372555bcea10a1242e0f721dbd36fc681039"} Nov 02 19:47:36.062026 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.061997 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" event={"ID":"58af3e1a-198c-4706-9bfa-e5b1ba476d4c","Type":"ContainerStarted","Data":"14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2"} Nov 02 19:47:36.062157 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.062044 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" event={"ID":"58af3e1a-198c-4706-9bfa-e5b1ba476d4c","Type":"ContainerStarted","Data":"59b02093da115b8bd1f152b888d52def3a15b795b9f729c04d6ca99230522351"} Nov 02 19:47:36.170857 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.170819 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:36.172431 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.172408 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 19:47:36.172560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.172461 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 19:47:36.183436 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.183371 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" podStartSLOduration=2.183353112 podStartE2EDuration="2.183353112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:36.180519993 +0000 UTC m=+1114.522909072" watchObservedRunningTime="2026-11-02 19:47:36.183353112 +0000 UTC m=+1114.525742193" Nov 02 19:47:36.315438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.315305 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" podStartSLOduration=2.3152812320000002 podStartE2EDuration="2.315281232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:36.305272916 +0000 UTC m=+1114.647662008" watchObservedRunningTime="2026-11-02 19:47:36.315281232 +0000 UTC m=+1114.657670312" Nov 02 19:47:36.439497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.438909 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podStartSLOduration=2.438885696 podStartE2EDuration="2.438885696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:36.435962916 +0000 UTC m=+1114.778352020" watchObservedRunningTime="2026-11-02 19:47:36.438885696 +0000 UTC m=+1114.781274779" Nov 02 19:47:36.904752 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.904718 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:36.925498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.906515 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 19:47:36.925498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.906581 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 19:47:36.970872 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.970836 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:36.980184 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.980146 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 19:47:36.980508 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:36.980454 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 19:47:37.179050 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:37.178953 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 19:47:37.179692 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:37.179656 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 19:47:37.905341 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:37.905306 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 19:47:37.905525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:37.905363 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 19:47:37.960498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:37.959943 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 19:47:37.960498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:37.960011 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 19:47:38.171560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:38.171050 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 19:47:38.171560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:38.171136 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 19:47:38.683496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:38.683192 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:38.905290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:38.905254 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 19:47:38.905511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:38.905307 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 19:47:38.970930 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:38.970821 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 19:47:38.970930 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:38.970891 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 19:47:39.170970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:39.170926 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 19:47:39.171251 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:39.171219 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 19:47:39.904688 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:39.904647 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 19:47:39.905122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:39.904706 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 19:47:39.959497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:39.958917 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 19:47:39.959497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:39.958984 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 19:47:40.175559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:40.175364 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 19:47:40.175559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:40.175423 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 19:47:40.904701 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:40.904666 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 19:47:40.905107 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:40.904718 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 19:47:40.974500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:40.957875 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 19:47:40.974500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:40.957940 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 19:47:41.175560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:41.171946 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 19:47:41.175560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:41.172014 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 19:47:41.905932 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:41.905897 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 19:47:41.906412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:41.905961 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 19:47:41.970185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:41.970146 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 19:47:41.970391 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:41.970211 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 19:47:42.170621 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:42.170522 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 19:47:42.170621 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:42.170597 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 19:47:43.286510 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.284828 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:43.287048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.287023 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:43.287124 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.287114 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 19:47:43.380402 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.379929 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:43.381570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.381086 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:43.382144 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.382099 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 19:47:43.498717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.498675 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:43.581607 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.576782 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:43.581607 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.578138 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:43.599219 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:43.599175 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 19:47:44.919279 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:47:44.919238 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:47:45.840686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:45.839719 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:47:45.998616 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:45.998565 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:47:46.015632 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:46.015576 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:47:46.017386 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:46.016780 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 02 19:47:46.017386 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:46.017117 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 19:47:46.095554 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:46.095112 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:47:46.361693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:46.361575 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:47:48.243790 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:48.243756 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:52.775955 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:52.775912 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:57.281573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:57.281486 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:58.504504 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:58.504458 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:47:58.670378 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:47:58.670338 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:01.732673 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:01.732625 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:04.941765 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:04.941729 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7"] Nov 02 19:48:04.967824 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:04.967785 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:04.993331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:04.993299 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b"] Nov 02 19:48:04.997611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:04.997575 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.027319 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.027282 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c753243f-3ba8-46c1-97d3-517915d494f3-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.027644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.027616 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.027895 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.027785 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.028050 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.028032 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c753243f-3ba8-46c1-97d3-517915d494f3-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.029201 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.029178 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.029367 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.029328 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.029537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.029374 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.029537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.029426 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h72hb\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-kube-api-access-h72hb\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.034979 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.029748 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.053602 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.053549 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7"] Nov 02 19:48:05.076301 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.076269 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b"] Nov 02 19:48:05.132544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132508 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c753243f-3ba8-46c1-97d3-517915d494f3-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.132728 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132567 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.132728 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132596 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.132728 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132621 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c753243f-3ba8-46c1-97d3-517915d494f3-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.132728 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132646 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.132728 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132667 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.132728 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132692 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.132728 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132723 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.133141 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132773 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h72hb\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-kube-api-access-h72hb\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.133141 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.132829 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.133141 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133129 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.133313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133168 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.133313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133186 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.133313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133196 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.133313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133239 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.133313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133259 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.133313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133276 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.133579 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133318 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c753243f-3ba8-46c1-97d3-517915d494f3-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.133579 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133409 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.133579 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133469 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.133674 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133648 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.133674 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133670 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2552h\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-kube-api-access-2552h\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.133813 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.133791 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.135056 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.135035 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.135165 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.135146 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c753243f-3ba8-46c1-97d3-517915d494f3-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.192271 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.192196 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.192410 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.192359 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72hb\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-kube-api-access-h72hb\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.235064 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235020 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235263 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235075 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235263 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235099 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235263 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235124 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235263 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235156 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235489 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235266 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235489 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235290 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2552h\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-kube-api-access-2552h\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235489 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235414 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235491 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235626 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.235879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235849 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.236005 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.235882 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.236071 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.236017 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.236071 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.236022 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.237919 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.237891 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.238079 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.238058 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.286945 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.286908 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:05.294285 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.294259 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.294416 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.294346 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2552h\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-kube-api-access-2552h\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.320414 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.320372 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:05.716826 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:48:05.716792 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc753243f_3ba8_46c1_97d3_517915d494f3.slice/crio-9033e8b4651a1da7385c56586cac79b4f077fc473681aebe97a40642adf761fe WatchSource:0}: Error finding container 9033e8b4651a1da7385c56586cac79b4f077fc473681aebe97a40642adf761fe: Status 404 returned error can't find the container with id 9033e8b4651a1da7385c56586cac79b4f077fc473681aebe97a40642adf761fe Nov 02 19:48:05.718725 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.718697 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7"] Nov 02 19:48:05.719948 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.719911 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:05.720024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.719988 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:05.720084 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.720057 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:05.723102 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.723081 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b"] Nov 02 19:48:05.767806 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:48:05.767773 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f750f1_28c2_4d1c_a92e_00bb449e1d3b.slice/crio-bbc16d6f04e2efdc85f1528cf9c09f3b5e6cff2563d5f64eab0714ec2e3d64a4 WatchSource:0}: Error finding container bbc16d6f04e2efdc85f1528cf9c09f3b5e6cff2563d5f64eab0714ec2e3d64a4: Status 404 returned error can't find the container with id bbc16d6f04e2efdc85f1528cf9c09f3b5e6cff2563d5f64eab0714ec2e3d64a4 Nov 02 19:48:05.770813 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.770780 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:05.770921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.770865 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:05.770986 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:05.770908 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:06.158140 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:06.158111 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:06.446053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:06.445965 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" event={"ID":"02f750f1-28c2-4d1c-a92e-00bb449e1d3b","Type":"ContainerStarted","Data":"84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e"} Nov 02 19:48:06.446053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:06.446030 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" event={"ID":"02f750f1-28c2-4d1c-a92e-00bb449e1d3b","Type":"ContainerStarted","Data":"bbc16d6f04e2efdc85f1528cf9c09f3b5e6cff2563d5f64eab0714ec2e3d64a4"} Nov 02 19:48:06.447880 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:06.447840 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" event={"ID":"c753243f-3ba8-46c1-97d3-517915d494f3","Type":"ContainerStarted","Data":"e5df1ab4987429c70c1150b58cb9b77f07708659bcf9184c24bec086e4a9e4c9"} Nov 02 19:48:06.448061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:06.447899 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" event={"ID":"c753243f-3ba8-46c1-97d3-517915d494f3","Type":"ContainerStarted","Data":"9033e8b4651a1da7385c56586cac79b4f077fc473681aebe97a40642adf761fe"} Nov 02 19:48:06.569695 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:06.569642 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" podStartSLOduration=2.569617454 podStartE2EDuration="2.569617454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:06.565632194 +0000 UTC m=+1144.908021276" watchObservedRunningTime="2026-11-02 19:48:06.569617454 +0000 UTC m=+1144.912006534" Nov 02 19:48:06.688276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:06.688220 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" podStartSLOduration=2.688204319 podStartE2EDuration="2.688204319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:06.688004015 +0000 UTC m=+1145.030393095" watchObservedRunningTime="2026-11-02 19:48:06.688204319 +0000 UTC m=+1145.030593399" Nov 02 19:48:07.288001 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:07.287965 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:07.289666 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:07.289638 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 19:48:07.289807 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:07.289697 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 19:48:07.327499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:07.322711 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:07.327499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:07.322860 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 19:48:07.327499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:07.322901 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 19:48:08.287860 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:08.287829 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 19:48:08.288069 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:08.288048 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 19:48:08.325500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:08.323899 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 19:48:08.325500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:08.323966 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 19:48:09.287954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:09.287910 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 19:48:09.288139 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:09.287979 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 19:48:09.368548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:09.366036 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 19:48:09.368548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:09.366131 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 19:48:09.797684 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:09.797651 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:10.287445 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:10.287414 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 19:48:10.287708 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:10.287684 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 19:48:10.318499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:10.317123 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:10.356501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:10.342684 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 19:48:10.356501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:10.342748 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 19:48:10.868012 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:10.867979 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:11.287665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:11.287571 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 19:48:11.287909 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:11.287878 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 19:48:11.363495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:11.362995 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 19:48:11.363495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:11.363056 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 19:48:12.287562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:12.287519 5850 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 19:48:12.287749 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:12.287579 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 19:48:12.327804 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:12.327353 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 19:48:12.327804 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:12.327435 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 19:48:13.364418 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:13.364383 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:13.366421 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:13.366393 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:13.367739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:13.367714 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:48:13.521813 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:13.521778 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:13.522540 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:13.522458 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:13.523920 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:13.523886 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:48:14.766875 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:48:14.766839 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:48:15.481059 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:15.481031 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:19.954301 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:19.954266 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979"] Nov 02 19:48:19.958693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:19.958668 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:19.961192 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:19.961156 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:20.036376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.005607 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44370a98-8413-404c-b45e-bf9bbc523424-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.036376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.019591 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.036376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.019661 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.036376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.019724 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxxg\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-kube-api-access-4vxxg\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.036376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.019768 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.036376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.019808 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.036376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.019980 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.036376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.020039 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.036376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.020092 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44370a98-8413-404c-b45e-bf9bbc523424-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.092061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.092020 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979"] Nov 02 19:48:20.121694 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.121661 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44370a98-8413-404c-b45e-bf9bbc523424-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.121878 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.121702 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.121878 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.121751 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.121878 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.121799 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxxg\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-kube-api-access-4vxxg\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.121878 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.121854 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.122013 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.121891 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.122047 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.122011 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.122082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.122047 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.122113 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.122093 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44370a98-8413-404c-b45e-bf9bbc523424-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.122635 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.122451 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44370a98-8413-404c-b45e-bf9bbc523424-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.123005 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.122972 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.123144 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.123087 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.123216 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.123147 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.123635 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.123608 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.125711 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.125689 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.126178 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.126152 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44370a98-8413-404c-b45e-bf9bbc523424-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.181672 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.181640 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.181831 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.181773 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxxg\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-kube-api-access-4vxxg\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.394499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.390265 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:20.940647 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.940593 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979"] Nov 02 19:48:20.980180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.980125 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:20.980683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.980231 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:20.980683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:20.980280 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:21.625732 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:21.625695 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" event={"ID":"44370a98-8413-404c-b45e-bf9bbc523424","Type":"ContainerStarted","Data":"79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631"} Nov 02 19:48:21.625982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:21.625963 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" event={"ID":"44370a98-8413-404c-b45e-bf9bbc523424","Type":"ContainerStarted","Data":"19dba6f9f1bf12dc2cad54d72242c9e229bd65944e9a3d9bd5d0c5f75c2efdda"} Nov 02 19:48:21.796512 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:21.796377 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" podStartSLOduration=2.796355933 podStartE2EDuration="2.796355933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:21.776259465 +0000 UTC m=+1160.118648547" watchObservedRunningTime="2026-11-02 19:48:21.796355933 +0000 UTC m=+1160.138745014" Nov 02 19:48:22.393506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:22.391576 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:22.394606 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:22.394019 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 19:48:22.394606 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:22.394077 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 19:48:23.391626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:23.391587 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 19:48:23.391805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:23.391644 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 19:48:24.391487 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:24.391441 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 19:48:24.391967 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:24.391533 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 19:48:24.676088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:24.676002 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:25.391647 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:25.391607 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 19:48:25.392061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:25.391683 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 19:48:26.391610 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:26.391575 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 19:48:26.391796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:26.391646 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 19:48:27.390965 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:27.390926 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 19:48:27.391169 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:27.390996 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 19:48:28.391763 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:28.391724 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 19:48:28.392166 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:28.391788 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 19:48:29.162619 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:29.162585 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:29.395583 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:29.395550 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:29.396071 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:29.395934 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:29.396723 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:29.396703 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 19:48:33.289866 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:33.289831 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:33.550343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:33.550248 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:38.035536 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:38.035491 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:42.468373 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:42.468291 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:44.761930 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:48:44.761888 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:48:46.940868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:46.940829 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:50.254593 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.254546 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7"] Nov 02 19:48:50.432777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.432737 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7"] Nov 02 19:48:50.434737 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.434709 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.436522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.436493 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfscd\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-kube-api-access-vfscd\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.437706 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.437683 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.437908 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.437888 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9144aefe-bea4-4000-bf7e-b2c016c2591e-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.438740 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.438719 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.438932 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.438912 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.439114 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.439097 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.439288 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.439270 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.439435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.439418 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.439925 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.439907 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.550778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.550677 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfscd\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-kube-api-access-vfscd\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.554596 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.554564 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.554842 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.554818 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9144aefe-bea4-4000-bf7e-b2c016c2591e-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.554978 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.554960 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.555103 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.555085 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.555243 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.555227 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.555376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.555360 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.570629 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.570582 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.574148 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.572140 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.574148 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.572822 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9144aefe-bea4-4000-bf7e-b2c016c2591e-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.574148 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.573059 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.574148 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.573704 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.574580 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.574558 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.575087 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.574951 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.627161 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.627123 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.639496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.630821 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfscd\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-kube-api-access-vfscd\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.669589 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.653586 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.669821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.665098 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:50.913778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:50.913736 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:51.001947 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:51.001899 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 19:48:51.017614 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:51.017563 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.018235 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:51.018201 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 02 19:48:51.032672 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:51.032627 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.113635 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:51.113578 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.299555 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:51.294598 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.364496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:51.354629 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.634848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:51.590578 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 19:48:51.856559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:51.815579 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.888905 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:48:51.888776 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9144aefe_bea4_4000_bf7e_b2c016c2591e.slice/crio-bd7c7d0f8d75eaeb74a047cc7d495f31e1e1fea6f7997b54faf5fe020f90b3e8 WatchSource:0}: Error finding container bd7c7d0f8d75eaeb74a047cc7d495f31e1e1fea6f7997b54faf5fe020f90b3e8: Status 404 returned error can't find the container with id bd7c7d0f8d75eaeb74a047cc7d495f31e1e1fea6f7997b54faf5fe020f90b3e8 Nov 02 19:48:52.001408 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.001363 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:52.001863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.001727 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:52.002283 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.002052 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:48:52.025704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.025642 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 19:48:52.039769 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.027827 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:52.039769 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.038690 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:52.087144 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.087102 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7"] Nov 02 19:48:52.087504 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.087464 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:48:52.105965 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.105937 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:48:52.107123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.107102 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:48:52.118600 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.118554 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 19:48:52.120082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.120054 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:48:52.362440 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.329209 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:52.510542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.376666 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:48:52.510542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.408568 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:52.510542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.409305 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:48:52.535665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.532869 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:48:52.535665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.533385 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:48:52.745400 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.697202 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:48:52.745400 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.726316 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:52.978500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:52.972160 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:48:53.039066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:53.038971 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 19:48:53.072496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:53.071273 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:48:53.080778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:53.080742 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:48:53.090496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:53.090029 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" event={"ID":"9144aefe-bea4-4000-bf7e-b2c016c2591e","Type":"ContainerStarted","Data":"bd7c7d0f8d75eaeb74a047cc7d495f31e1e1fea6f7997b54faf5fe020f90b3e8"} Nov 02 19:48:54.097977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:54.097936 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" event={"ID":"9144aefe-bea4-4000-bf7e-b2c016c2591e","Type":"ContainerStarted","Data":"adc5b08ab4e78fbaac501d6c7b901ec21a829a4645aebe989f88d625ae186920"} Nov 02 19:48:54.234524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:54.234426 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" podStartSLOduration=5.234408434 podStartE2EDuration="5.234408434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:54.220982402 +0000 UTC m=+1192.563371507" watchObservedRunningTime="2026-11-02 19:48:54.234408434 +0000 UTC m=+1192.576797513" Nov 02 19:48:54.931756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:54.931719 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:48:54.933058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:54.933021 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 19:48:54.933215 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:54.933087 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 19:48:55.932326 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:55.929981 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 19:48:55.932326 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:55.930055 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 19:48:56.923792 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:56.923754 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 19:48:56.924100 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:56.924071 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 19:48:57.253946 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:57.253862 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:57.926868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:57.926832 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 19:48:57.927175 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:57.927147 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 19:48:58.931659 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:58.931623 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 19:48:58.932239 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:48:58.931961 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 19:49:00.357504 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:00.357435 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:49:00.357977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:00.357777 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:49:00.358856 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:00.358831 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:49:01.750124 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:01.750092 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:06.349911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:06.349855 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:08.780945 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:08.780908 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:10.243734 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:10.243694 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:10.709561 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:10.709527 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:11.008571 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:11.007332 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:15.035501 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:49:15.033165 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:49:15.688637 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:15.688576 5850 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:15.689290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:15.689259 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 19:49:15.706635 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:15.706582 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 19:49:15.806648 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:15.806593 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:15.984651 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:15.984545 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 19:49:16.000866 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.000816 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 19:49:16.082625 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.082572 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.083666 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.083636 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 19:49:16.543785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.543732 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 02 19:49:16.544443 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.544028 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.711496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.707784 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.711496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.709331 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:49:16.750390 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.745596 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.750390 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.746080 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:49:16.822214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.820574 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 19:49:16.822214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:16.821040 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:49:17.129082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.129036 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:17.132129 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.132106 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:49:17.136497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.133128 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:17.136497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.136153 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:49:17.159492 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.147121 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:49:17.168872 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.167960 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:17.199848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.199786 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:17.218513 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.218063 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:49:17.281071 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.281040 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:49:17.360569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.360535 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:49:17.431097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.430993 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:49:17.432137 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.432099 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:49:17.459348 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:17.459311 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:49:19.957846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:19.957810 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh"] Nov 02 19:49:19.962204 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:19.962183 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.048779 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.048733 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh"] Nov 02 19:49:20.065080 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.065037 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.065274 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.065111 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.065274 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.065211 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.065378 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.065356 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.065436 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.065404 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.065548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.065525 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjmfz\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-kube-api-access-hjmfz\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.065596 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.065565 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.065680 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.065628 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.065775 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.065761 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167262 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167224 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167262 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167270 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167503 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167293 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167503 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167392 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167503 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167433 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167675 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167467 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjmfz\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-kube-api-access-hjmfz\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167675 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167529 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167675 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167592 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167675 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167649 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167724 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.167879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.167821 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.168030 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.168006 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.168030 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.168016 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.168544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.168519 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.169765 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.169743 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.170081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.170059 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.226463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.226385 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.226721 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.226691 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjmfz\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-kube-api-access-hjmfz\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.275863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.275827 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:20.697423 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:49:20.697377 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568ce497_2bab_4ae1_a9cd_3c386dbc28e6.slice/crio-52c733ac778157d9db0720452e72fe5547eba4e5f5e4a9d5300b87b6dcab37a4 WatchSource:0}: Error finding container 52c733ac778157d9db0720452e72fe5547eba4e5f5e4a9d5300b87b6dcab37a4: Status 404 returned error can't find the container with id 52c733ac778157d9db0720452e72fe5547eba4e5f5e4a9d5300b87b6dcab37a4 Nov 02 19:49:20.700293 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.700250 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:20.700384 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.700343 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:20.700429 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.700392 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:20.700883 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:20.700856 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh"] Nov 02 19:49:21.399945 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:21.399911 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" event={"ID":"568ce497-2bab-4ae1-a9cd-3c386dbc28e6","Type":"ContainerStarted","Data":"a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1"} Nov 02 19:49:21.400371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:21.399965 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" event={"ID":"568ce497-2bab-4ae1-a9cd-3c386dbc28e6","Type":"ContainerStarted","Data":"52c733ac778157d9db0720452e72fe5547eba4e5f5e4a9d5300b87b6dcab37a4"} Nov 02 19:49:21.637288 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:21.637214 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" podStartSLOduration=2.637185927 podStartE2EDuration="2.637185927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:21.576040658 +0000 UTC m=+1219.918429738" watchObservedRunningTime="2026-11-02 19:49:21.637185927 +0000 UTC m=+1219.979575075" Nov 02 19:49:21.660506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:21.660389 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:21.853496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:21.830542 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:22.276409 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:22.276373 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:22.277759 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:22.277733 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 19:49:22.277834 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:22.277785 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 19:49:22.472646 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:22.472607 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:22.723453 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:22.723420 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:23.285496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.272801 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn"] Nov 02 19:49:23.329521 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.314162 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 19:49:23.329521 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.314225 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 19:49:23.465551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.457522 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:23.571525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.571290 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn"] Nov 02 19:49:23.571525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.571417 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.631494 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.627628 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d23a1e27-openshift-default-dockercfg-vjqkf\"" Nov 02 19:49:23.663359 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.663321 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:23.773007 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.772909 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-credential-socket\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.775207 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.775173 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-socket\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.775537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.775504 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-certs\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.775762 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.775743 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-data\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.775928 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.775912 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-envoy\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.776196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.776180 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4ce348-3fa9-441f-ae13-40998c61314e-istiod-ca-cert\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.798844 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.796162 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-podinfo\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.798844 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.798654 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp2g7\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-kube-api-access-jp2g7\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.799106 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.798851 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-token\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900116 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900070 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-token\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900175 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-credential-socket\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900213 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-socket\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900261 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-certs\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900302 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-data\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900340 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-envoy\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900449 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4ce348-3fa9-441f-ae13-40998c61314e-istiod-ca-cert\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900567 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-podinfo\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900660 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp2g7\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-kube-api-access-jp2g7\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900717 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-socket\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.900985 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.900965 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-credential-socket\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.901581 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.901559 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-data\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.901711 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.901618 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-certs\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.902115 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.902095 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4ce348-3fa9-441f-ae13-40998c61314e-istiod-ca-cert\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.903095 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.903075 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-envoy\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.903661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.903639 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-podinfo\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.959265 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.959231 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-token\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:23.959439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:23.959287 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp2g7\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-kube-api-access-jp2g7\") pod \"e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:24.227725 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:24.227687 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:24.277942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:24.277851 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 19:49:24.277942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:24.277920 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 19:49:24.654417 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:49:24.654387 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4ce348_3fa9_441f_ae13_40998c61314e.slice/crio-12d42a99f1622965cecb6a05d83f29c9abcad516a828ae93a91490e324459153 WatchSource:0}: Error finding container 12d42a99f1622965cecb6a05d83f29c9abcad516a828ae93a91490e324459153: Status 404 returned error can't find the container with id 12d42a99f1622965cecb6a05d83f29c9abcad516a828ae93a91490e324459153 Nov 02 19:49:24.657053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:24.657017 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:24.657164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:24.657088 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:24.657164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:24.657115 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:24.657726 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:24.657704 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn"] Nov 02 19:49:25.276976 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:25.276935 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 19:49:25.277196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:25.276996 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 19:49:25.591090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:25.590995 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" event={"ID":"8a4ce348-3fa9-441f-ae13-40998c61314e","Type":"ContainerStarted","Data":"a40db9bdfd22e9da82e6c99015f4f35ab11d9ff6cbd749761b1cf7a529e47584"} Nov 02 19:49:25.591090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:25.591046 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" event={"ID":"8a4ce348-3fa9-441f-ae13-40998c61314e","Type":"ContainerStarted","Data":"12d42a99f1622965cecb6a05d83f29c9abcad516a828ae93a91490e324459153"} Nov 02 19:49:25.834372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:25.834253 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" podStartSLOduration=2.834239198 podStartE2EDuration="2.834239198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:25.761351396 +0000 UTC m=+1224.103740477" watchObservedRunningTime="2026-11-02 19:49:25.834239198 +0000 UTC m=+1224.176628278" Nov 02 19:49:26.231594 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:26.231544 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:26.237045 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:26.237016 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:26.277094 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:26.277060 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 19:49:26.277260 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:26.277123 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 19:49:26.547921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:26.547824 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:26.640143 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:26.640108 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:26.649840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:26.649774 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:49:27.276584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:27.276546 5850 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 19:49:27.277082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:27.276612 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 19:49:27.746565 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:27.746515 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:27.925515 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:27.925459 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:28.094938 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:28.094852 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:28.466902 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:28.466865 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:28.467595 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:28.467571 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:28.468576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:28.468558 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:49:28.958565 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:28.958522 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:29.110805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:29.110768 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:29.276541 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:29.276432 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:29.434381 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:29.434348 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:29.606619 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:29.606524 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:30.390783 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.390745 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b"] Nov 02 19:49:30.397445 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.397416 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.456743 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.456706 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6d4e9ef2-openshift-default-dockercfg-gkf7z\"" Nov 02 19:49:30.628509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.625936 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4vw\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-kube-api-access-ll4vw\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.628509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.626086 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-podinfo\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.628509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.626172 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-certs\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.628509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.626259 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-envoy\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.628509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.626288 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-socket\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.628509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.626317 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-credential-socket\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.628509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.626409 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d65f4-05cf-48ed-a7f5-570a3c11a675-istiod-ca-cert\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.628509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.626451 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-token\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.628509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.626556 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-data\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.667065 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.666966 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:30.681591 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.681552 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b"] Nov 02 19:49:30.727615 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.727574 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-certs\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.727882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.727644 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-envoy\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.727882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.727674 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-socket\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.727882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.727698 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-credential-socket\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.727882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.727782 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d65f4-05cf-48ed-a7f5-570a3c11a675-istiod-ca-cert\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.727882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.727818 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-token\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.727882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.727879 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-data\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.728214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.727965 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4vw\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-kube-api-access-ll4vw\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.728214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.728059 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-podinfo\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.728214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.728192 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-credential-socket\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.728394 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.728358 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-data\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.728394 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.728379 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-certs\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.728586 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.728434 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-socket\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.728586 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.728538 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d65f4-05cf-48ed-a7f5-570a3c11a675-istiod-ca-cert\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.730159 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.730132 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-envoy\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.730572 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.730550 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-podinfo\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.792108 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.792076 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-token\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.792295 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.792252 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4vw\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-kube-api-access-ll4vw\") pod \"e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:30.877340 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:30.877306 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:31.108100 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:31.108010 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:31.158611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:31.158535 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:31.358592 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:31.358280 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:31.970208 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:31.967807 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:31.970208 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:31.967920 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:31.970208 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:31.967963 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:31.971650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:31.971581 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b"] Nov 02 19:49:32.039781 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:32.039751 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:32.790152 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:32.790117 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" event={"ID":"950d65f4-05cf-48ed-a7f5-570a3c11a675","Type":"ContainerStarted","Data":"e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954"} Nov 02 19:49:32.790372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:32.790171 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" event={"ID":"950d65f4-05cf-48ed-a7f5-570a3c11a675","Type":"ContainerStarted","Data":"07faf471400c8818fe8e25a75cb6bd931489b9a213804fdfec56a4a827581f91"} Nov 02 19:49:32.962753 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:32.962696 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" podStartSLOduration=2.9626813370000002 podStartE2EDuration="2.962681337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:32.961276416 +0000 UTC m=+1231.303665497" watchObservedRunningTime="2026-11-02 19:49:32.962681337 +0000 UTC m=+1231.305070416" Nov 02 19:49:33.125532 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:33.122400 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:33.199591 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:33.197307 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:33.796456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:33.796427 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:33.797623 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:33.797602 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:49:34.862933 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.862882 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr"] Nov 02 19:49:34.869706 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.868127 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.926605 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.926564 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.926777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.926625 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.926777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.926756 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.926869 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.926802 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b79v8\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-kube-api-access-b79v8\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.926918 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.926867 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.926971 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.926935 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.927107 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.927087 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad146ede-1b0e-4d32-b7bd-c95590295f34-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.927214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.927197 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.927268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.927228 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:34.989202 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.989155 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr"] Nov 02 19:49:34.989400 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.989254 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x"] Nov 02 19:49:34.989880 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.989522 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" containerID="cri-o://dbc0746e2401f934a8a4521a1236f1a8b1d8ddfca71f32697ea81e55214e49f5" gracePeriod=30 Nov 02 19:49:34.996957 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.996933 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr"] Nov 02 19:49:34.997142 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:34.997071 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.019803 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.019773 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:35.028865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.028829 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.029016 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.028950 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.029071 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029034 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.029138 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029117 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad146ede-1b0e-4d32-b7bd-c95590295f34-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.029178 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029161 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxnv\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-kube-api-access-nnxnv\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.029243 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029228 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.029308 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029293 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.029350 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029314 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.029350 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029321 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.029671 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029626 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.029799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029780 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.029852 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029824 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.029904 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029861 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.029904 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029894 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.030003 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029920 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.030003 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.029970 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.030210 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.030000 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad146ede-1b0e-4d32-b7bd-c95590295f34-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.030210 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.030005 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.030777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.030752 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.031002 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.030981 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.031123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.031101 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b79v8\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-kube-api-access-b79v8\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.031194 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.031154 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.031259 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.031225 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc4e519-23d7-4081-89d1-4f69cc466f26-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.032906 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.032865 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.037835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.033732 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.060553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.041794 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x"] Nov 02 19:49:35.099280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.098852 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.099280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.099088 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b79v8\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-kube-api-access-b79v8\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.133527 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.133454 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.133704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.133558 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.133704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.133591 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.133704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.133646 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.133867 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.133733 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc4e519-23d7-4081-89d1-4f69cc466f26-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.133924 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.133876 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.133924 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.133914 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.134007 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.133918 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.134007 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.133934 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.134099 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.134072 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxnv\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-kube-api-access-nnxnv\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.134157 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.134143 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.134214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.134174 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.134268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.134219 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.134516 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.134453 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc4e519-23d7-4081-89d1-4f69cc466f26-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.136362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.136339 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.136463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.136406 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.185118 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.185084 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:35.198313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.198247 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxnv\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-kube-api-access-nnxnv\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.198465 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.198389 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.198863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.198837 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:35.328085 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.327952 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:35.569682 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:49:35.569648 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad146ede_1b0e_4d32_b7bd_c95590295f34.slice/crio-73b22f87a564ab0a48aeba4ea7c1c8327fce050a7b054f6c7e9ffdba355ffa26 WatchSource:0}: Error finding container 73b22f87a564ab0a48aeba4ea7c1c8327fce050a7b054f6c7e9ffdba355ffa26: Status 404 returned error can't find the container with id 73b22f87a564ab0a48aeba4ea7c1c8327fce050a7b054f6c7e9ffdba355ffa26 Nov 02 19:49:35.572462 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.572425 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:35.572588 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.572568 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:35.572645 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.572620 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:35.573029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.573007 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr"] Nov 02 19:49:35.706420 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:49:35.706382 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc4e519_23d7_4081_89d1_4f69cc466f26.slice/crio-f5a1c7d179c050cacda0710dcfccd3671d2927bc70ca8dec1c4ca4285c143fff WatchSource:0}: Error finding container f5a1c7d179c050cacda0710dcfccd3671d2927bc70ca8dec1c4ca4285c143fff: Status 404 returned error can't find the container with id f5a1c7d179c050cacda0710dcfccd3671d2927bc70ca8dec1c4ca4285c143fff Nov 02 19:49:35.709642 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.709611 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x"] Nov 02 19:49:35.709844 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.709816 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:35.709896 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.709883 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:35.709937 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.709915 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:49:35.818704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.818665 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" event={"ID":"ad146ede-1b0e-4d32-b7bd-c95590295f34","Type":"ContainerStarted","Data":"77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f"} Nov 02 19:49:35.818890 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.818720 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" event={"ID":"ad146ede-1b0e-4d32-b7bd-c95590295f34","Type":"ContainerStarted","Data":"73b22f87a564ab0a48aeba4ea7c1c8327fce050a7b054f6c7e9ffdba355ffa26"} Nov 02 19:49:35.819343 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.819310 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" event={"ID":"9fc4e519-23d7-4081-89d1-4f69cc466f26","Type":"ContainerStarted","Data":"c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110"} Nov 02 19:49:35.819545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.819358 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" event={"ID":"9fc4e519-23d7-4081-89d1-4f69cc466f26","Type":"ContainerStarted","Data":"f5a1c7d179c050cacda0710dcfccd3671d2927bc70ca8dec1c4ca4285c143fff"} Nov 02 19:49:35.935075 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:35.934955 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podStartSLOduration=1.934934711 podStartE2EDuration="1.934934711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:35.932441904 +0000 UTC m=+1234.274830999" watchObservedRunningTime="2026-11-02 19:49:35.934934711 +0000 UTC m=+1234.277323792" Nov 02 19:49:36.186091 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:36.186000 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:36.187928 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:36.187898 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 19:49:36.187994 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:36.187965 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 19:49:36.333057 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:36.332896 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:36.335330 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:36.335290 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 19:49:36.335694 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:36.335648 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 19:49:36.372355 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:36.372319 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:36.762290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:36.762256 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:36.962128 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:36.962050 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:37.162520 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:37.162408 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:37.187019 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:37.186557 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 19:49:37.187019 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:37.186615 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 19:49:37.330445 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:37.330408 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 19:49:37.330769 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:37.330737 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 19:49:38.185490 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:38.185433 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 19:49:38.185899 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:38.185505 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 19:49:38.341658 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:38.330895 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 19:49:38.341658 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:38.330971 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 19:49:39.054498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:39.053598 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:39.272505 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:39.248050 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:39.299747 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:39.299714 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 19:49:39.386756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:39.386716 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 19:49:39.472501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:39.377155 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 19:49:39.472501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:39.461310 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 19:49:39.955501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:39.943511 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podStartSLOduration=5.943465842 podStartE2EDuration="5.943465842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:36.057970202 +0000 UTC m=+1234.400359282" watchObservedRunningTime="2026-11-02 19:49:39.943465842 +0000 UTC m=+1238.285854926" Nov 02 19:49:40.206559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.203696 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.283834 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.227724 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 19:49:40.283834 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.227806 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 19:49:40.392337 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.390832 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 19:49:40.392723 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.392684 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 19:49:40.392986 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.392951 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 19:49:40.443539 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.410674 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh"] Nov 02 19:49:40.482043 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.462875 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.486457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.482606 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.503512 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.496833 5850 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.511544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.510725 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh"] Nov 02 19:49:40.511544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.510863 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:40.566520 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.556230 5850 generic.go:356] "Generic (PLEG): container finished" podID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerID="dbc0746e2401f934a8a4521a1236f1a8b1d8ddfca71f32697ea81e55214e49f5" exitCode=0 Nov 02 19:49:40.566520 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.556296 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" event={"ID":"8ad964b2-ea0b-4893-abc2-1376231e0e50","Type":"ContainerDied","Data":"dbc0746e2401f934a8a4521a1236f1a8b1d8ddfca71f32697ea81e55214e49f5"} Nov 02 19:49:40.568704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.567610 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.581033 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.579844 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.592950 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.590243 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsssb\" (UniqueName: \"kubernetes.io/projected/5b5beb63-12ef-49f0-84b2-e02c617dccc7-kube-api-access-fsssb\") pod \"payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh\" (UID: \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:40.592950 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.590440 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5beb63-12ef-49f0-84b2-e02c617dccc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh\" (UID: \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:40.638979 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.638901 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d8ffab69\"" Nov 02 19:49:40.639425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.639366 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d8ffab69-dockercfg-982tx\"" Nov 02 19:49:40.692741 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.692701 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsssb\" (UniqueName: \"kubernetes.io/projected/5b5beb63-12ef-49f0-84b2-e02c617dccc7-kube-api-access-fsssb\") pod \"payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh\" (UID: \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:40.692954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.692861 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5beb63-12ef-49f0-84b2-e02c617dccc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh\" (UID: \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:40.693828 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.693802 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5beb63-12ef-49f0-84b2-e02c617dccc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh\" (UID: \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:40.768139 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.768027 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsssb\" (UniqueName: \"kubernetes.io/projected/5b5beb63-12ef-49f0-84b2-e02c617dccc7-kube-api-access-fsssb\") pod \"payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh\" (UID: \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:40.880328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.880291 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:49:40.952142 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.952103 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:40.996771 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.996737 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-token\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-token\") pod \"8ad964b2-ea0b-4893-abc2-1376231e0e50\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " Nov 02 19:49:40.996965 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.996796 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-credential-socket\") pod \"8ad964b2-ea0b-4893-abc2-1376231e0e50\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " Nov 02 19:49:40.996965 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.996854 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-podinfo\") pod \"8ad964b2-ea0b-4893-abc2-1376231e0e50\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " Nov 02 19:49:40.996965 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.996949 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-socket\") pod \"8ad964b2-ea0b-4893-abc2-1376231e0e50\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " Nov 02 19:49:40.997126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.997033 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-kube-api-access-8cc79\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-kube-api-access-8cc79\") pod \"8ad964b2-ea0b-4893-abc2-1376231e0e50\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " Nov 02 19:49:40.997179 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.997165 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-certs\") pod \"8ad964b2-ea0b-4893-abc2-1376231e0e50\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " Nov 02 19:49:40.997233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.997168 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-credential-socket" pod "8ad964b2-ea0b-4893-abc2-1376231e0e50" (UID: "8ad964b2-ea0b-4893-abc2-1376231e0e50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:49:40.997233 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.997206 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad964b2-ea0b-4893-abc2-1376231e0e50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad964b2-ea0b-4893-abc2-1376231e0e50-istiod-ca-cert\") pod \"8ad964b2-ea0b-4893-abc2-1376231e0e50\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " Nov 02 19:49:40.997351 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.997328 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-data\") pod \"8ad964b2-ea0b-4893-abc2-1376231e0e50\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " Nov 02 19:49:40.997422 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.997367 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-envoy\") pod \"8ad964b2-ea0b-4893-abc2-1376231e0e50\" (UID: \"8ad964b2-ea0b-4893-abc2-1376231e0e50\") " Nov 02 19:49:40.997628 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.997603 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-socket" pod "8ad964b2-ea0b-4893-abc2-1376231e0e50" (UID: "8ad964b2-ea0b-4893-abc2-1376231e0e50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:49:40.998068 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.998039 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad964b2-ea0b-4893-abc2-1376231e0e50-istiod-ca-cert" pod "8ad964b2-ea0b-4893-abc2-1376231e0e50" (UID: "8ad964b2-ea0b-4893-abc2-1376231e0e50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:40.998184 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.998147 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:40.998184 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.998169 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:40.998297 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.998268 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-certs" pod "8ad964b2-ea0b-4893-abc2-1376231e0e50" (UID: "8ad964b2-ea0b-4893-abc2-1376231e0e50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:49:40.998371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.998349 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-data" pod "8ad964b2-ea0b-4893-abc2-1376231e0e50" (UID: "8ad964b2-ea0b-4893-abc2-1376231e0e50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:49:40.999523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.999493 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-podinfo" pod "8ad964b2-ea0b-4893-abc2-1376231e0e50" (UID: "8ad964b2-ea0b-4893-abc2-1376231e0e50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:49:40.999773 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:40.999750 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-envoy" pod "8ad964b2-ea0b-4893-abc2-1376231e0e50" (UID: "8ad964b2-ea0b-4893-abc2-1376231e0e50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:49:41.000029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.000006 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-token" pod "8ad964b2-ea0b-4893-abc2-1376231e0e50" (UID: "8ad964b2-ea0b-4893-abc2-1376231e0e50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:41.000575 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.000550 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-kube-api-access-8cc79" pod "8ad964b2-ea0b-4893-abc2-1376231e0e50" (UID: "8ad964b2-ea0b-4893-abc2-1376231e0e50"). InnerVolumeSpecName "kube-api-access-8cc79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:41.100176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.100135 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:41.100176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.100163 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:41.100176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.100173 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:41.100176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.100181 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad964b2-ea0b-4893-abc2-1376231e0e50-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:41.100539 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.100191 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cc79\" (UniqueName: \"kubernetes.io/projected/8ad964b2-ea0b-4893-abc2-1376231e0e50-kube-api-access-8cc79\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:41.100539 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.100202 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad964b2-ea0b-4893-abc2-1376231e0e50-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:41.100539 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.100211 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad964b2-ea0b-4893-abc2-1376231e0e50-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:41.186495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.185756 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 19:49:41.186495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.185805 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 19:49:41.331375 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.330860 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 19:49:41.331375 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.330926 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 19:49:41.399062 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.399033 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh"] Nov 02 19:49:41.573636 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.573605 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" Nov 02 19:49:41.573817 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.573631 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr" event={"ID":"8ad964b2-ea0b-4893-abc2-1376231e0e50","Type":"ContainerDied","Data":"6db8cda59922715e88a1420e806bb74abad55e61b5733387cc43cb4771220e9e"} Nov 02 19:49:41.573817 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.573698 5850 scope.go:122] "RemoveContainer" containerID="dbc0746e2401f934a8a4521a1236f1a8b1d8ddfca71f32697ea81e55214e49f5" Nov 02 19:49:41.581788 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.581721 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" event={"ID":"5b5beb63-12ef-49f0-84b2-e02c617dccc7","Type":"ContainerStarted","Data":"4c9bab958bb5e5b7e41e559dbc4daa56f4a63380a5fc3b6e30ee29303737766a"} Nov 02 19:49:41.770992 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.770952 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr"] Nov 02 19:49:41.774203 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.774171 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-xwqbr"] Nov 02 19:49:41.882880 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:41.882849 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" path="/var/lib/kubelet/pods/8ad964b2-ea0b-4893-abc2-1376231e0e50/volumes" Nov 02 19:49:42.224560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.201835 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 19:49:42.224560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.201896 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 19:49:42.276279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.276218 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:42.333418 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.333374 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 19:49:42.333956 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.333441 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 19:49:42.471559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.459251 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:42.686313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.686235 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" event={"ID":"5b5beb63-12ef-49f0-84b2-e02c617dccc7","Type":"ContainerStarted","Data":"7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a"} Nov 02 19:49:42.688717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.688691 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:42.881496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.841869 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg"] Nov 02 19:49:42.881496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.842968 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" Nov 02 19:49:42.881496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.842990 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" Nov 02 19:49:42.881496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.843163 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad964b2-ea0b-4893-abc2-1376231e0e50" containerName="istio-proxy" Nov 02 19:49:42.881496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.864143 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:42.934086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.934049 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d86b4de-f2e5-460d-b512-35694007416d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg\" (UID: \"5d86b4de-f2e5-460d-b512-35694007416d\") " pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:42.960566 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.960458 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdsg\" (UniqueName: \"kubernetes.io/projected/5d86b4de-f2e5-460d-b512-35694007416d-kube-api-access-tgdsg\") pod \"payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg\" (UID: \"5d86b4de-f2e5-460d-b512-35694007416d\") " pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:42.965498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.942007 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6d4e9ef2\"" Nov 02 19:49:42.977307 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:42.942767 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6d4e9ef2-dockercfg-ntk2r\"" Nov 02 19:49:43.176500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.142450 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d86b4de-f2e5-460d-b512-35694007416d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg\" (UID: \"5d86b4de-f2e5-460d-b512-35694007416d\") " pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:43.176500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.142566 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdsg\" (UniqueName: \"kubernetes.io/projected/5d86b4de-f2e5-460d-b512-35694007416d-kube-api-access-tgdsg\") pod \"payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg\" (UID: \"5d86b4de-f2e5-460d-b512-35694007416d\") " pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:43.176500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.144241 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d86b4de-f2e5-460d-b512-35694007416d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg\" (UID: \"5d86b4de-f2e5-460d-b512-35694007416d\") " pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:43.176500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.144298 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg"] Nov 02 19:49:43.514549 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.484440 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" podStartSLOduration=4.484418324 podStartE2EDuration="4.484418324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:42.94224355 +0000 UTC m=+1241.284632630" watchObservedRunningTime="2026-11-02 19:49:43.484418324 +0000 UTC m=+1241.826807408" Nov 02 19:49:43.514549 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.492594 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 19:49:43.514549 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.492653 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 19:49:43.514549 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.505835 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 19:49:43.514549 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.505899 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 19:49:43.529554 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.529515 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:43.536147 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.536112 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdsg\" (UniqueName: \"kubernetes.io/projected/5d86b4de-f2e5-460d-b512-35694007416d-kube-api-access-tgdsg\") pod \"payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg\" (UID: \"5d86b4de-f2e5-460d-b512-35694007416d\") " pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:43.545643 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.542053 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv"] Nov 02 19:49:43.545643 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.544143 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:43.585329 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.585297 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv"] Nov 02 19:49:43.585679 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.585659 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:43.589702 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.589670 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvmlp\" (UniqueName: \"kubernetes.io/projected/17112c9c-c35c-4466-b034-1a9ab330f5c5-kube-api-access-nvmlp\") pod \"payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv\" (UID: \"17112c9c-c35c-4466-b034-1a9ab330f5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:43.590093 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.590065 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17112c9c-c35c-4466-b034-1a9ab330f5c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv\" (UID: \"17112c9c-c35c-4466-b034-1a9ab330f5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:43.692708 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.692652 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvmlp\" (UniqueName: \"kubernetes.io/projected/17112c9c-c35c-4466-b034-1a9ab330f5c5-kube-api-access-nvmlp\") pod \"payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv\" (UID: \"17112c9c-c35c-4466-b034-1a9ab330f5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:43.692909 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.692800 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17112c9c-c35c-4466-b034-1a9ab330f5c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv\" (UID: \"17112c9c-c35c-4466-b034-1a9ab330f5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:43.693871 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.693830 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17112c9c-c35c-4466-b034-1a9ab330f5c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv\" (UID: \"17112c9c-c35c-4466-b034-1a9ab330f5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:43.800514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.791101 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvmlp\" (UniqueName: \"kubernetes.io/projected/17112c9c-c35c-4466-b034-1a9ab330f5c5-kube-api-access-nvmlp\") pod \"payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv\" (UID: \"17112c9c-c35c-4466-b034-1a9ab330f5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:43.924503 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:43.922850 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:44.008015 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.007982 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg"] Nov 02 19:49:44.409661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.409634 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv"] Nov 02 19:49:44.551828 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.551789 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:44.552678 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.552657 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:44.553329 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.553310 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 19:49:44.706812 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.706732 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:44.707125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.707103 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:44.708712 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.708681 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 19:49:44.715232 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.715197 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" event={"ID":"17112c9c-c35c-4466-b034-1a9ab330f5c5","Type":"ContainerStarted","Data":"55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546"} Nov 02 19:49:44.715320 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.715239 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" event={"ID":"17112c9c-c35c-4466-b034-1a9ab330f5c5","Type":"ContainerStarted","Data":"7f186b8e2fdc86957f5baed82d67de141153007a38894c5a26ef92fa4d7ddc5d"} Nov 02 19:49:44.715320 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.715273 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:44.717117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.717094 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" event={"ID":"5d86b4de-f2e5-460d-b512-35694007416d","Type":"ContainerStarted","Data":"1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b"} Nov 02 19:49:44.717271 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.717253 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" event={"ID":"5d86b4de-f2e5-460d-b512-35694007416d","Type":"ContainerStarted","Data":"353b3ad3f21ddf19eb19b8c6bea6d2386eddc1e1a318c257d5de4eef6b6e20e0"} Nov 02 19:49:44.717494 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:44.717466 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:44.802225 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:49:44.802181 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:49:45.035045 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:45.034936 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" podStartSLOduration=3.03492259 podStartE2EDuration="3.03492259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:45.034584968 +0000 UTC m=+1243.376974061" watchObservedRunningTime="2026-11-02 19:49:45.03492259 +0000 UTC m=+1243.377311670" Nov 02 19:49:46.263853 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:46.263786 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:46.440578 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:46.440540 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:47.091701 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:47.091657 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:48.455522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:48.445545 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:49.392926 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:49.392893 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:49.687995 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:49.687912 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:49.853718 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:49.853682 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:52.946261 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:52.946221 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:54.763512 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:54.759709 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:49:54.880499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:54.880132 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" podStartSLOduration=12.880110783 podStartE2EDuration="12.880110783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:45.1872189 +0000 UTC m=+1243.529607982" watchObservedRunningTime="2026-11-02 19:49:54.880110783 +0000 UTC m=+1253.222499863" Nov 02 19:49:55.528808 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:55.528771 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:55.731827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:55.731790 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:55.768730 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:55.768695 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:49:56.144499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.141216 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 19:49:56.169514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.153087 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:56.303499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.300407 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.362497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.362165 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.459798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.459649 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.477435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.477301 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.516610 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.516561 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 19:49:56.537634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.537582 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.557658 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.537957 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 02 19:49:56.749319 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:56.749201 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:49:57.157548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.151813 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.157548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.152605 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:49:57.174811 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.172549 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.388042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.387994 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.388996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.388970 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:49:57.415174 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.415071 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.416560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.416538 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:49:57.504974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.504927 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.517499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.507086 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 19:49:57.518212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.518181 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:49:57.522148 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.522124 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:49:57.557792 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.557744 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 02 19:49:57.638859 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.519176 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 19:49:57.639781 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.639739 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.663525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.647333 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:49:57.978865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.978836 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:49:57.979145 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:57.979127 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:49:58.185444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.185394 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:58.191850 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.186148 5850 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:58.192267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.192232 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:58.442891 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.421053 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 19:49:58.446455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.446402 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 19:49:58.508422 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.507224 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 19:49:58.518274 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.518236 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:58.887137 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.887091 5850 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:58.893897 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.893857 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 02 19:49:58.895103 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.895055 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:58.897317 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:58.896014 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 19:49:59.050128 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.048704 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:59.055221 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:49:59.055179 5850 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.076s" Nov 02 19:49:59.055518 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.055500 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:49:59.091522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.091491 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" Nov 02 19:49:59.092267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.092224 5850 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 02 19:49:59.095653 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.095618 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 19:49:59.115360 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.115330 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 19:49:59.132450 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.117338 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 19:49:59.137353 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.137273 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 19:49:59.170273 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.170235 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 19:49:59.172330 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.172300 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 19:49:59.174703 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.174682 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 19:49:59.178552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.178510 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 19:49:59.217091 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.217063 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:49:59.389645 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:49:59.389567 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:02.145517 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:02.145407 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:02.320642 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:02.320555 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:03.744811 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:03.744782 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:04.912685 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:04.912650 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:05.543717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:05.543681 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:07.900319 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:07.900287 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf"] Nov 02 19:50:07.906193 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:07.906165 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:07.961429 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:07.961392 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf"] Nov 02 19:50:07.992594 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:07.992559 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7134c5c2-347a-4420-ba32-bd7d49741c59-openshift-service-ca\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:07.992765 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:07.992732 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:07.992837 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:07.992822 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmhn\" (UniqueName: \"kubernetes.io/projected/7134c5c2-347a-4420-ba32-bd7d49741c59-kube-api-access-bwmhn\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:07.992953 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:07.992937 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-metrics-certs\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.009597 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.009562 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d23a1e27-serving-cert\"" Nov 02 19:50:08.094842 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.094799 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7134c5c2-347a-4420-ba32-bd7d49741c59-openshift-service-ca\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.095052 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.094914 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.095052 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.094976 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmhn\" (UniqueName: \"kubernetes.io/projected/7134c5c2-347a-4420-ba32-bd7d49741c59-kube-api-access-bwmhn\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.095184 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:50:08.095060 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-serving-cert: secret "maas-api-e2e-ait-d23a1e27-serving-cert" not found Nov 02 19:50:08.095184 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:50:08.095126 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls podName:7134c5c2-347a-4420-ba32-bd7d49741c59 nodeName:}" failed. No retries permitted until 2026-11-02 19:50:08.595108862 +0000 UTC m=+1266.937497924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls") pod "maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" (UID: "7134c5c2-347a-4420-ba32-bd7d49741c59") : secret "maas-api-e2e-ait-d23a1e27-serving-cert" not found Nov 02 19:50:08.095184 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.095065 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-metrics-certs\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.095599 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.095579 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7134c5c2-347a-4420-ba32-bd7d49741c59-openshift-service-ca\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.097470 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.097446 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-metrics-certs\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.103130 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.103104 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:08.185547 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.184970 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmhn\" (UniqueName: \"kubernetes.io/projected/7134c5c2-347a-4420-ba32-bd7d49741c59-kube-api-access-bwmhn\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.258409 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.258365 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:08.415223 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.415191 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:08.605671 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.605577 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.608088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.608055 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls\") pod \"maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:08.820725 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:08.820681 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:09.200859 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:09.200829 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf"] Nov 02 19:50:09.201635 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:50:09.201592 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7134c5c2_347a_4420_ba32_bd7d49741c59.slice/crio-eb4b19a42a53e82f400b5508cb5f25c0925485c4ca843f5ecc225e4895833fbf WatchSource:0}: Error finding container eb4b19a42a53e82f400b5508cb5f25c0925485c4ca843f5ecc225e4895833fbf: Status 404 returned error can't find the container with id eb4b19a42a53e82f400b5508cb5f25c0925485c4ca843f5ecc225e4895833fbf Nov 02 19:50:09.202545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:09.202524 5850 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:50:09.238431 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:09.238390 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" event={"ID":"7134c5c2-347a-4420-ba32-bd7d49741c59","Type":"ContainerStarted","Data":"eb4b19a42a53e82f400b5508cb5f25c0925485c4ca843f5ecc225e4895833fbf"} Nov 02 19:50:09.294654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:09.294621 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:09.460234 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:09.460121 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:10.190828 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:10.190796 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:10.246641 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:10.246605 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" event={"ID":"7134c5c2-347a-4420-ba32-bd7d49741c59","Type":"ContainerStarted","Data":"2d38ab58692f35950a3042b6c571193ef3adf6308fea6ad60b6df3fe16675329"} Nov 02 19:50:10.247056 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:10.246666 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:10.384868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:10.384803 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" podStartSLOduration=2.507140955 podStartE2EDuration="3.384786707s" totalImagesPullingTime="877.645752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:50:10.382563612 +0000 UTC m=+1268.724952686" watchObservedRunningTime="2026-11-02 19:50:10.384786707 +0000 UTC m=+1268.727175788" Nov 02 19:50:10.613220 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:10.613141 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:12.837016 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:12.836982 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:14.763506 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:50:14.763454 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:50:16.258930 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:16.258900 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:50:17.238624 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:17.238593 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:19.931831 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:19.931794 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6"] Nov 02 19:50:19.936305 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:19.936282 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:19.983413 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:19.983377 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr"] Nov 02 19:50:19.987694 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:19.987675 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.037901 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.037858 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6"] Nov 02 19:50:20.097930 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.097890 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71596876-6e7f-47d5-8972-ea460f0fa1f6-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.098127 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.097954 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aca617-dd63-4b17-8151-a864f0b23aab-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.098127 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098090 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.098214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098139 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.098254 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098210 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.098288 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098279 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.098403 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098386 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.098439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098415 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.098544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098526 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqmjq\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-kube-api-access-pqmjq\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.098601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098556 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.098649 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098598 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.098703 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098649 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.098703 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098677 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.098804 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098713 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.098804 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098736 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.098804 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098776 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.098941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098822 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz2vp\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-kube-api-access-qz2vp\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.098941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.098866 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.102364 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.102330 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr"] Nov 02 19:50:20.200697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200611 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.200697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200648 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.200915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200717 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqmjq\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-kube-api-access-pqmjq\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.200915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200741 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.200915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200773 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.200915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200821 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.200915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200850 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.200915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200885 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.200915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200910 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.201279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200946 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.201279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.200984 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz2vp\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-kube-api-access-qz2vp\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.201279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201015 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.201279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201091 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71596876-6e7f-47d5-8972-ea460f0fa1f6-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.201279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201109 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.201279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201226 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.201617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201280 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.201617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201333 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aca617-dd63-4b17-8151-a864f0b23aab-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.201617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201389 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.201617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201393 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.201617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201421 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.201617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201520 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.201617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201538 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.202022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201624 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.202022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201748 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.202022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201818 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.202022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.201980 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71596876-6e7f-47d5-8972-ea460f0fa1f6-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.202232 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.202114 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.202232 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.202148 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aca617-dd63-4b17-8151-a864f0b23aab-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.203609 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.203577 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.203780 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.203762 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.203850 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.203814 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.204153 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.204135 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.259466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.259428 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.259466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.259499 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.259875 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.259857 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz2vp\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-kube-api-access-qz2vp\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.260003 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.259986 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqmjq\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-kube-api-access-pqmjq\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.301999 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.301962 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:20.550369 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.550281 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:20.723927 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:50:20.723898 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71596876_6e7f_47d5_8972_ea460f0fa1f6.slice/crio-a0be621d23e63791aa97c8791b6fbad07e37bb365df84e03d22399cc166666cb WatchSource:0}: Error finding container a0be621d23e63791aa97c8791b6fbad07e37bb365df84e03d22399cc166666cb: Status 404 returned error can't find the container with id a0be621d23e63791aa97c8791b6fbad07e37bb365df84e03d22399cc166666cb Nov 02 19:50:20.726916 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.726881 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:50:20.727027 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.726965 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:50:20.727091 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.727031 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:50:20.727510 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.727487 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr"] Nov 02 19:50:20.925527 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:50:20.925496 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5aca617_dd63_4b17_8151_a864f0b23aab.slice/crio-405b0e674d78ff1ad3e903295107606585c0bf5d6fe2eabff9bd0c559b0aa138 WatchSource:0}: Error finding container 405b0e674d78ff1ad3e903295107606585c0bf5d6fe2eabff9bd0c559b0aa138: Status 404 returned error can't find the container with id 405b0e674d78ff1ad3e903295107606585c0bf5d6fe2eabff9bd0c559b0aa138 Nov 02 19:50:20.926598 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.926573 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6"] Nov 02 19:50:20.928174 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.928141 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:50:20.928224 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.928205 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:50:20.928255 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:20.928236 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:50:21.313807 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.313710 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84"] Nov 02 19:50:21.319060 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.319030 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.359493 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.359445 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" event={"ID":"71596876-6e7f-47d5-8972-ea460f0fa1f6","Type":"ContainerStarted","Data":"c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771"} Nov 02 19:50:21.359677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.359523 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" event={"ID":"71596876-6e7f-47d5-8972-ea460f0fa1f6","Type":"ContainerStarted","Data":"a0be621d23e63791aa97c8791b6fbad07e37bb365df84e03d22399cc166666cb"} Nov 02 19:50:21.361286 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.361260 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" event={"ID":"e5aca617-dd63-4b17-8151-a864f0b23aab","Type":"ContainerStarted","Data":"02c5de160dcc0ee0dd8580e6d2de7f870cf1d6d53132e5d6017bc3e2eabc04a9"} Nov 02 19:50:21.361411 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.361304 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" event={"ID":"e5aca617-dd63-4b17-8151-a864f0b23aab","Type":"ContainerStarted","Data":"405b0e674d78ff1ad3e903295107606585c0bf5d6fe2eabff9bd0c559b0aa138"} Nov 02 19:50:21.373454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.373085 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d8ffab69-serving-cert\"" Nov 02 19:50:21.374950 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.374920 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84"] Nov 02 19:50:21.441111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.439693 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.441111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.439756 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-metrics-certs\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.441111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.439880 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c27b02-deed-4ed8-aed5-b6d134c071c7-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.441111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.440059 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj24l\" (UniqueName: \"kubernetes.io/projected/57c27b02-deed-4ed8-aed5-b6d134c071c7-kube-api-access-dj24l\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.535557 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.535517 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:21.541923 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.541881 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c27b02-deed-4ed8-aed5-b6d134c071c7-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.542123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.541989 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj24l\" (UniqueName: \"kubernetes.io/projected/57c27b02-deed-4ed8-aed5-b6d134c071c7-kube-api-access-dj24l\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.544206 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.542328 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.544206 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.542395 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-metrics-certs\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.544206 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:50:21.542422 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-serving-cert: secret "maas-api-e2e-migrate-d8ffab69-serving-cert" not found Nov 02 19:50:21.544206 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:50:21.542525 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls podName:57c27b02-deed-4ed8-aed5-b6d134c071c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:50:22.042467265 +0000 UTC m=+1280.384856328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls") pod "maas-api-e2e-migrate-d8ffab69-546485d64-cff84" (UID: "57c27b02-deed-4ed8-aed5-b6d134c071c7") : secret "maas-api-e2e-migrate-d8ffab69-serving-cert" not found Nov 02 19:50:21.544206 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.542923 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c27b02-deed-4ed8-aed5-b6d134c071c7-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.558664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.558392 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:21.558664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.558548 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 19:50:21.558664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.558599 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 19:50:21.642591 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.631467 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-metrics-certs\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.658499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.649468 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj24l\" (UniqueName: \"kubernetes.io/projected/57c27b02-deed-4ed8-aed5-b6d134c071c7-kube-api-access-dj24l\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:21.715692 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.711574 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:21.739500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.735254 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" podStartSLOduration=2.735233105 podStartE2EDuration="2.735233105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:50:21.586770754 +0000 UTC m=+1279.929159859" watchObservedRunningTime="2026-11-02 19:50:21.735233105 +0000 UTC m=+1280.077622187" Nov 02 19:50:21.739500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.735728 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" podStartSLOduration=2.735716509 podStartE2EDuration="2.735716509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:50:21.711115207 +0000 UTC m=+1280.053504298" watchObservedRunningTime="2026-11-02 19:50:21.735716509 +0000 UTC m=+1280.078105591" Nov 02 19:50:21.888500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:21.878674 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:22.073358 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:22.073254 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:22.088362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:22.088336 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls\") pod \"maas-api-e2e-migrate-d8ffab69-546485d64-cff84\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:22.257810 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:22.257756 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:22.304289 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:22.303949 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:22.305794 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:22.305757 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 19:50:22.305922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:22.305820 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 19:50:22.559021 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:22.558976 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 19:50:22.559544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:22.559038 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 19:50:22.675589 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:22.675563 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84"] Nov 02 19:50:23.304017 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:23.303972 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 19:50:23.304317 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:23.304276 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 19:50:23.392910 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:23.392859 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" event={"ID":"57c27b02-deed-4ed8-aed5-b6d134c071c7","Type":"ContainerStarted","Data":"6d97015e2616dcd32bde857cb8779be8f2e04be0c34ae03fa486baea3ba6a2d6"} Nov 02 19:50:23.561887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:23.561768 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 19:50:23.561887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:23.561823 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 19:50:24.305497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:24.304939 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 19:50:24.305497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:24.305006 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 19:50:24.574412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:24.571073 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 19:50:24.574412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:24.571140 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 19:50:24.578463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:24.578423 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" event={"ID":"57c27b02-deed-4ed8-aed5-b6d134c071c7","Type":"ContainerStarted","Data":"c562096d922725304df696e6e7eb02ae0a538edcc185429d00ea48bcd139cc6e"} Nov 02 19:50:24.580425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:24.580351 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:24.734497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:24.733537 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" podStartSLOduration=2.9060771340000002 podStartE2EDuration="3.733516765s" totalImagesPullingTime="827.439631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:50:24.731955687 +0000 UTC m=+1283.074344778" watchObservedRunningTime="2026-11-02 19:50:24.733516765 +0000 UTC m=+1283.075905849" Nov 02 19:50:25.303173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.303133 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 19:50:25.303362 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.303187 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 19:50:25.551498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.550896 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 19:50:25.551498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.550966 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 19:50:25.725500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.706462 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r"] Nov 02 19:50:25.738499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.734756 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:25.769526 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.769461 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r"] Nov 02 19:50:25.789705 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.789671 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6d4e9ef2-serving-cert\"" Nov 02 19:50:25.891698 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.891660 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce179b1-545b-497a-a471-2f18c4698b9a-openshift-service-ca\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:25.891891 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.891724 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4jw\" (UniqueName: \"kubernetes.io/projected/5ce179b1-545b-497a-a471-2f18c4698b9a-kube-api-access-jm4jw\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:25.891891 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.891862 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-metrics-certs\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:25.892093 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.892073 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-maas-api-tls\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:25.996550 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:25.990200 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:26.030705 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.030659 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-metrics-certs\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:26.038740 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.038669 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-maas-api-tls\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:26.077501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.071548 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce179b1-545b-497a-a471-2f18c4698b9a-openshift-service-ca\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:26.077501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.071756 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4jw\" (UniqueName: \"kubernetes.io/projected/5ce179b1-545b-497a-a471-2f18c4698b9a-kube-api-access-jm4jw\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:26.077501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.074117 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-maas-api-tls\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:26.097514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.089547 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce179b1-545b-497a-a471-2f18c4698b9a-openshift-service-ca\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:26.102514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.102079 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-metrics-certs\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:26.147916 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.147321 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:26.156558 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.155941 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4jw\" (UniqueName: \"kubernetes.io/projected/5ce179b1-545b-497a-a471-2f18c4698b9a-kube-api-access-jm4jw\") pod \"maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:26.305202 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.305096 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 19:50:26.305504 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.305440 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 19:50:26.384139 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.384104 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:26.562066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.561971 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 19:50:26.562066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.562038 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 19:50:26.893098 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:50:26.893063 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce179b1_545b_497a_a471_2f18c4698b9a.slice/crio-63e1a56d07585dae0ca149e0fed674d3cc28f6a4bd257c5b333b3c71660a6827 WatchSource:0}: Error finding container 63e1a56d07585dae0ca149e0fed674d3cc28f6a4bd257c5b333b3c71660a6827: Status 404 returned error can't find the container with id 63e1a56d07585dae0ca149e0fed674d3cc28f6a4bd257c5b333b3c71660a6827 Nov 02 19:50:26.895392 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:26.895334 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r"] Nov 02 19:50:27.303086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:27.302993 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 19:50:27.303086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:27.303060 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 19:50:27.550872 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:27.550832 5850 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 19:50:27.551078 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:27.550897 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 19:50:27.611593 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:27.611303 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" event={"ID":"5ce179b1-545b-497a-a471-2f18c4698b9a","Type":"ContainerStarted","Data":"63e1a56d07585dae0ca149e0fed674d3cc28f6a4bd257c5b333b3c71660a6827"} Nov 02 19:50:28.709502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:28.695340 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" event={"ID":"5ce179b1-545b-497a-a471-2f18c4698b9a","Type":"ContainerStarted","Data":"25a4f5942cb220397a7cbabaace0a171b4595c9bb10ece6e90497ddd7275d504"} Nov 02 19:50:28.709502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:28.696935 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:28.855511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:28.844922 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" podStartSLOduration=3.037851273 podStartE2EDuration="3.844896606s" totalImagesPullingTime="807.045333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:50:28.825388595 +0000 UTC m=+1287.167777693" watchObservedRunningTime="2026-11-02 19:50:28.844896606 +0000 UTC m=+1287.187285690" Nov 02 19:50:28.985553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:28.979514 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:28.985553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:28.981413 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:29.017855 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:29.017817 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:50:29.193042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:29.192996 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:29.193546 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:29.193410 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:29.194899 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:29.194880 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:50:30.591016 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:30.590987 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:50:30.632599 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:30.632569 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:35.133673 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:35.133640 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:35.742109 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:35.742075 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:50:38.963457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:38.963410 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:39.611958 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:39.611928 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:41.850618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:41.850579 5850 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:50:41.851237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:41.851203 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:50:42.270172 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:42.270101 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:44.449947 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:44.449914 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:44.762718 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:50:44.762629 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:50:48.842384 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:48.842351 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:53.404748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:53.404682 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:50:57.737451 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:50:57.737388 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:02.238800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:02.238752 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:04.868316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.868272 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp"] Nov 02 19:51:04.873762 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.873735 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.930930 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.930892 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qmwl4\"" Nov 02 19:51:04.933730 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.933263 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.933995 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.933976 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czrh\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-kube-api-access-4czrh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.934231 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.934213 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba8694-51fd-4387-85b2-808b7ebbdc93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.934825 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.934792 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.934957 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.934927 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.935091 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.935055 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.935225 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.935203 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.935300 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.935256 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.935359 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.935341 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:04.988448 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:04.988409 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp"] Nov 02 19:51:05.037498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.037428 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.037812 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.037784 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czrh\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-kube-api-access-4czrh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.038036 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.038017 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba8694-51fd-4387-85b2-808b7ebbdc93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.038237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.038222 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.038409 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.038395 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.038600 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.038583 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.038750 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.038736 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.038868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.038856 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.038988 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.038975 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.039134 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.039010 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba8694-51fd-4387-85b2-808b7ebbdc93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.039325 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.039302 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.039523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.039487 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.040052 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.040012 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.040253 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.040094 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.054733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.054697 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.055404 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.055380 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.096422 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.096389 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czrh\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-kube-api-access-4czrh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.102935 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.102902 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-lv6rp\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.187760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.187662 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:05.645248 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:51:05.645217 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ba8694_51fd_4387_85b2_808b7ebbdc93.slice/crio-bf96d543a9a2e06429128a5504e71e0e1b6250997b806e0144885677d0693955 WatchSource:0}: Error finding container bf96d543a9a2e06429128a5504e71e0e1b6250997b806e0144885677d0693955: Status 404 returned error can't find the container with id bf96d543a9a2e06429128a5504e71e0e1b6250997b806e0144885677d0693955 Nov 02 19:51:05.646735 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.646708 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp"] Nov 02 19:51:05.648009 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.647978 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:51:05.648075 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.648044 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:51:05.648112 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:05.648073 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:51:06.063889 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:06.063800 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" event={"ID":"58ba8694-51fd-4387-85b2-808b7ebbdc93","Type":"ContainerStarted","Data":"2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70"} Nov 02 19:51:06.063889 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:06.063867 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" event={"ID":"58ba8694-51fd-4387-85b2-808b7ebbdc93","Type":"ContainerStarted","Data":"bf96d543a9a2e06429128a5504e71e0e1b6250997b806e0144885677d0693955"} Nov 02 19:51:06.184249 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:06.184193 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podStartSLOduration=2.18417723 podStartE2EDuration="2.18417723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:06.183375496 +0000 UTC m=+1324.525764580" watchObservedRunningTime="2026-11-02 19:51:06.18417723 +0000 UTC m=+1324.526566306" Nov 02 19:51:06.187989 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:06.187952 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:06.190110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:06.190068 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 19:51:06.190256 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:06.190150 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 19:51:07.188273 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:07.188234 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 19:51:07.188783 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:07.188290 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 19:51:07.199608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:07.199571 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:08.189337 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:08.189300 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 19:51:08.189756 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:08.189352 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 19:51:09.189051 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:09.189009 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 19:51:09.189235 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:09.189087 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 19:51:10.188994 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:10.188948 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 19:51:10.189551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:10.189012 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 19:51:10.939513 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:10.939461 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:11.188342 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:11.188297 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 19:51:11.188599 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:11.188367 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 19:51:11.370332 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:11.370293 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:12.064203 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:12.064173 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:12.190440 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:12.189956 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 19:51:12.190440 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:12.190023 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 19:51:12.298042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:12.298006 5850 patch_prober.go:29] interesting pod/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": read tcp 10.134.0.2:58406->10.134.0.75:15021: read: connection reset by peer" start-of-body= Nov 02 19:51:12.298185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:12.298090 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": read tcp 10.134.0.2:58406->10.134.0.75:15021: read: connection reset by peer" Nov 02 19:51:13.188089 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:13.188047 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 19:51:13.188553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:13.188115 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 19:51:13.227781 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:13.227748 5850 generic.go:356] "Generic (PLEG): container finished" podID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerID="a40db9bdfd22e9da82e6c99015f4f35ab11d9ff6cbd749761b1cf7a529e47584" exitCode=0 Nov 02 19:51:13.227951 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:13.227825 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" event={"ID":"8a4ce348-3fa9-441f-ae13-40998c61314e","Type":"ContainerDied","Data":"a40db9bdfd22e9da82e6c99015f4f35ab11d9ff6cbd749761b1cf7a529e47584"} Nov 02 19:51:13.228286 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:13.228267 5850 scope.go:122] "RemoveContainer" containerID="a40db9bdfd22e9da82e6c99015f4f35ab11d9ff6cbd749761b1cf7a529e47584" Nov 02 19:51:13.229848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:13.229819 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:51:13.229904 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:13.229893 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:51:13.229942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:13.229925 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:51:14.254255 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:14.254220 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" event={"ID":"8a4ce348-3fa9-441f-ae13-40998c61314e","Type":"ContainerStarted","Data":"e7bf97075ef6c8dd336c0a98f6fd5fb125575cb6f704f69ace35cd2cba4fb0e3"} Nov 02 19:51:14.558817 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:14.558648 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:14.559558 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:14.559530 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:14.560254 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:14.560228 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 19:51:14.773243 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:51:14.773178 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:51:15.229955 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:15.229917 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:51:15.231629 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:15.231491 5850 patch_prober.go:29] interesting pod/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 19:51:15.231782 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:15.231660 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 19:51:16.232007 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:16.231943 5850 patch_prober.go:29] interesting pod/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 19:51:16.232455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:16.232022 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 19:51:16.366039 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:16.366005 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:17.230979 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:17.230935 5850 patch_prober.go:29] interesting pod/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 19:51:17.231288 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:17.231257 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 19:51:18.231646 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:18.231596 5850 patch_prober.go:29] interesting pod/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 19:51:18.232083 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:18.231686 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 19:51:18.698215 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:18.698181 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:18.905511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:18.876926 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:19.230958 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:19.230919 5850 patch_prober.go:29] interesting pod/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 19:51:19.231271 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:19.231243 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 19:51:20.686530 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:20.686453 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:51:20.687406 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:20.687355 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:51:20.697840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:20.697763 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:51:21.264899 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:21.264868 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:26.190634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:26.190597 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:29.015744 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:29.015704 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:29.181851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:29.181808 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:31.051204 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:31.051169 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:31.220226 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:31.220190 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:33.736224 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:33.736182 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg"] Nov 02 19:51:33.863826 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:33.863790 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv"] Nov 02 19:51:34.139890 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:34.139845 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:34.312079 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:34.312028 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:35.564712 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:35.564674 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:36.584850 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:36.584812 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh"] Nov 02 19:51:36.859253 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:36.859163 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:37.177889 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:37.177607 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:37.333297 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:37.333259 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:38.745498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:38.738600 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" podUID="5d86b4de-f2e5-460d-b512-35694007416d" containerName="payload-processing" containerID="cri-o://1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b" gracePeriod=25 Nov 02 19:51:38.897001 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:38.895885 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" podUID="17112c9c-c35c-4466-b034-1a9ab330f5c5" containerName="payload-pre-processing" containerID="cri-o://55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546" gracePeriod=25 Nov 02 19:51:39.348506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.347401 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:51:39.398560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.398529 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:51:39.536554 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.535694 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17112c9c-c35c-4466-b034-1a9ab330f5c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17112c9c-c35c-4466-b034-1a9ab330f5c5-plugins-config-volume\") pod \"17112c9c-c35c-4466-b034-1a9ab330f5c5\" (UID: \"17112c9c-c35c-4466-b034-1a9ab330f5c5\") " Nov 02 19:51:39.536554 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.535839 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d86b4de-f2e5-460d-b512-35694007416d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d86b4de-f2e5-460d-b512-35694007416d-plugins-config-volume\") pod \"5d86b4de-f2e5-460d-b512-35694007416d\" (UID: \"5d86b4de-f2e5-460d-b512-35694007416d\") " Nov 02 19:51:39.536554 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.535923 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d86b4de-f2e5-460d-b512-35694007416d-kube-api-access-tgdsg\" (UniqueName: \"kubernetes.io/projected/5d86b4de-f2e5-460d-b512-35694007416d-kube-api-access-tgdsg\") pod \"5d86b4de-f2e5-460d-b512-35694007416d\" (UID: \"5d86b4de-f2e5-460d-b512-35694007416d\") " Nov 02 19:51:39.536554 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.535961 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17112c9c-c35c-4466-b034-1a9ab330f5c5-kube-api-access-nvmlp\" (UniqueName: \"kubernetes.io/projected/17112c9c-c35c-4466-b034-1a9ab330f5c5-kube-api-access-nvmlp\") pod \"17112c9c-c35c-4466-b034-1a9ab330f5c5\" (UID: \"17112c9c-c35c-4466-b034-1a9ab330f5c5\") " Nov 02 19:51:39.537330 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.537106 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d86b4de-f2e5-460d-b512-35694007416d-plugins-config-volume" pod "5d86b4de-f2e5-460d-b512-35694007416d" (UID: "5d86b4de-f2e5-460d-b512-35694007416d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:39.537330 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.537301 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17112c9c-c35c-4466-b034-1a9ab330f5c5-plugins-config-volume" pod "17112c9c-c35c-4466-b034-1a9ab330f5c5" (UID: "17112c9c-c35c-4466-b034-1a9ab330f5c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:39.539941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.539913 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17112c9c-c35c-4466-b034-1a9ab330f5c5-kube-api-access-nvmlp" pod "17112c9c-c35c-4466-b034-1a9ab330f5c5" (UID: "17112c9c-c35c-4466-b034-1a9ab330f5c5"). InnerVolumeSpecName "kube-api-access-nvmlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:39.541076 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.541049 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d86b4de-f2e5-460d-b512-35694007416d-kube-api-access-tgdsg" pod "5d86b4de-f2e5-460d-b512-35694007416d" (UID: "5d86b4de-f2e5-460d-b512-35694007416d"). InnerVolumeSpecName "kube-api-access-tgdsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:39.633498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.633092 5850 generic.go:356] "Generic (PLEG): container finished" podID="17112c9c-c35c-4466-b034-1a9ab330f5c5" containerID="55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546" exitCode=0 Nov 02 19:51:39.633498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.633190 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" event={"ID":"17112c9c-c35c-4466-b034-1a9ab330f5c5","Type":"ContainerDied","Data":"55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546"} Nov 02 19:51:39.633498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.633246 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" event={"ID":"17112c9c-c35c-4466-b034-1a9ab330f5c5","Type":"ContainerDied","Data":"7f186b8e2fdc86957f5baed82d67de141153007a38894c5a26ef92fa4d7ddc5d"} Nov 02 19:51:39.633498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.633279 5850 scope.go:122] "RemoveContainer" containerID="55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546" Nov 02 19:51:39.633498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.633450 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv" Nov 02 19:51:39.639182 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.639059 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17112c9c-c35c-4466-b034-1a9ab330f5c5-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:39.639182 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.639095 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d86b4de-f2e5-460d-b512-35694007416d-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:39.639182 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.639109 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgdsg\" (UniqueName: \"kubernetes.io/projected/5d86b4de-f2e5-460d-b512-35694007416d-kube-api-access-tgdsg\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:39.639182 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.639123 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvmlp\" (UniqueName: \"kubernetes.io/projected/17112c9c-c35c-4466-b034-1a9ab330f5c5-kube-api-access-nvmlp\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:39.641523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.640080 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" Nov 02 19:51:39.641523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.640114 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" event={"ID":"5d86b4de-f2e5-460d-b512-35694007416d","Type":"ContainerDied","Data":"1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b"} Nov 02 19:51:39.641762 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.641722 5850 generic.go:356] "Generic (PLEG): container finished" podID="5d86b4de-f2e5-460d-b512-35694007416d" containerID="1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b" exitCode=0 Nov 02 19:51:39.641914 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.641840 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg" event={"ID":"5d86b4de-f2e5-460d-b512-35694007416d","Type":"ContainerDied","Data":"353b3ad3f21ddf19eb19b8c6bea6d2386eddc1e1a318c257d5de4eef6b6e20e0"} Nov 02 19:51:39.651843 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.651822 5850 scope.go:122] "RemoveContainer" containerID="55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546" Nov 02 19:51:39.652229 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:51:39.652202 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546\": container with ID starting with 55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546 not found: ID does not exist" containerID="55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546" Nov 02 19:51:39.652366 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.652235 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546"} err="failed to get container status \"55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546\": rpc error: code = NotFound desc = could not find container \"55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546\": container with ID starting with 55d8bca41482c901183cf39996faeb9681ee5df85e66d4dcd9a00d3607734546 not found: ID does not exist" Nov 02 19:51:39.652366 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.652264 5850 scope.go:122] "RemoveContainer" containerID="1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b" Nov 02 19:51:39.664690 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.664666 5850 scope.go:122] "RemoveContainer" containerID="1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b" Nov 02 19:51:39.665075 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:51:39.665040 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b\": container with ID starting with 1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b not found: ID does not exist" containerID="1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b" Nov 02 19:51:39.665157 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.665086 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b"} err="failed to get container status \"1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b\": rpc error: code = NotFound desc = could not find container \"1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b\": container with ID starting with 1df2214f55e13614b4484000da546cf26a069b06a553ba58ccbdc52e60ab8e6b not found: ID does not exist" Nov 02 19:51:39.845432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.845340 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg"] Nov 02 19:51:39.845432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.845387 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d4e9ef2-85f976f9fc-tf2fg"] Nov 02 19:51:39.848081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.848056 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:39.936461 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.936422 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv"] Nov 02 19:51:39.939695 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:39.939669 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d4e9ef2-6cd78d4f95-2qtwv"] Nov 02 19:51:40.038488 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:40.038433 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:40.879381 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:40.879341 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:41.586196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:41.586155 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" podUID="5b5beb63-12ef-49f0-84b2-e02c617dccc7" containerName="payload-pre-processing" containerID="cri-o://7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a" gracePeriod=25 Nov 02 19:51:41.822087 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:41.822053 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17112c9c-c35c-4466-b034-1a9ab330f5c5" path="/var/lib/kubelet/pods/17112c9c-c35c-4466-b034-1a9ab330f5c5/volumes" Nov 02 19:51:41.822583 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:41.822560 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d86b4de-f2e5-460d-b512-35694007416d" path="/var/lib/kubelet/pods/5d86b4de-f2e5-460d-b512-35694007416d/volumes" Nov 02 19:51:41.917327 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:41.917297 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:51:41.978936 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:41.978848 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b5beb63-12ef-49f0-84b2-e02c617dccc7-kube-api-access-fsssb\" (UniqueName: \"kubernetes.io/projected/5b5beb63-12ef-49f0-84b2-e02c617dccc7-kube-api-access-fsssb\") pod \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\" (UID: \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\") " Nov 02 19:51:41.979083 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:41.979054 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b5beb63-12ef-49f0-84b2-e02c617dccc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5beb63-12ef-49f0-84b2-e02c617dccc7-plugins-config-volume\") pod \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\" (UID: \"5b5beb63-12ef-49f0-84b2-e02c617dccc7\") " Nov 02 19:51:41.979484 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:41.979449 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5beb63-12ef-49f0-84b2-e02c617dccc7-plugins-config-volume" pod "5b5beb63-12ef-49f0-84b2-e02c617dccc7" (UID: "5b5beb63-12ef-49f0-84b2-e02c617dccc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:41.979751 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:41.979732 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5beb63-12ef-49f0-84b2-e02c617dccc7-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:41.981529 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:41.981506 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5beb63-12ef-49f0-84b2-e02c617dccc7-kube-api-access-fsssb" pod "5b5beb63-12ef-49f0-84b2-e02c617dccc7" (UID: "5b5beb63-12ef-49f0-84b2-e02c617dccc7"). InnerVolumeSpecName "kube-api-access-fsssb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:42.081582 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.081139 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsssb\" (UniqueName: \"kubernetes.io/projected/5b5beb63-12ef-49f0-84b2-e02c617dccc7-kube-api-access-fsssb\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:42.679762 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.679721 5850 generic.go:356] "Generic (PLEG): container finished" podID="5b5beb63-12ef-49f0-84b2-e02c617dccc7" containerID="7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a" exitCode=0 Nov 02 19:51:42.679966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.679764 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" event={"ID":"5b5beb63-12ef-49f0-84b2-e02c617dccc7","Type":"ContainerDied","Data":"7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a"} Nov 02 19:51:42.679966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.679779 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" Nov 02 19:51:42.679966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.679878 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh" event={"ID":"5b5beb63-12ef-49f0-84b2-e02c617dccc7","Type":"ContainerDied","Data":"4c9bab958bb5e5b7e41e559dbc4daa56f4a63380a5fc3b6e30ee29303737766a"} Nov 02 19:51:42.679966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.679914 5850 scope.go:122] "RemoveContainer" containerID="7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a" Nov 02 19:51:42.695767 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.695740 5850 scope.go:122] "RemoveContainer" containerID="7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a" Nov 02 19:51:42.696116 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:51:42.696095 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a\": container with ID starting with 7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a not found: ID does not exist" containerID="7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a" Nov 02 19:51:42.696210 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.696126 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a"} err="failed to get container status \"7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a\": rpc error: code = NotFound desc = could not find container \"7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a\": container with ID starting with 7ebb09690dd5f578a094e4efbd29a93197e3b70e4f1ec8f596d6186f7835819a not found: ID does not exist" Nov 02 19:51:42.849944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.849909 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh"] Nov 02 19:51:42.900643 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:42.900608 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8ffab69-74664b6459-n9bqh"] Nov 02 19:51:43.819014 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:43.818983 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b5beb63-12ef-49f0-84b2-e02c617dccc7" path="/var/lib/kubelet/pods/5b5beb63-12ef-49f0-84b2-e02c617dccc7/volumes" Nov 02 19:51:44.426051 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:44.426006 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:44.763246 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:51:44.763149 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:51:45.705636 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:45.705602 5850 generic.go:356] "Generic (PLEG): container finished" podID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerID="e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954" exitCode=0 Nov 02 19:51:45.706080 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:45.705676 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" event={"ID":"950d65f4-05cf-48ed-a7f5-570a3c11a675","Type":"ContainerDied","Data":"e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954"} Nov 02 19:51:45.706212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:45.706195 5850 scope.go:122] "RemoveContainer" containerID="e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954" Nov 02 19:51:45.707920 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:45.707892 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:51:45.707988 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:45.707961 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:51:45.708024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:45.708008 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:51:46.735698 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:46.735662 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" event={"ID":"950d65f4-05cf-48ed-a7f5-570a3c11a675","Type":"ContainerStarted","Data":"b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6"} Nov 02 19:51:46.771945 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:46.771907 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:47.131497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:47.131346 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:51:47.131497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:47.131463 5850 patch_prober.go:29] interesting pod/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 19:51:47.131802 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:47.131548 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 19:51:48.129501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.125557 5850 patch_prober.go:29] interesting pod/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 19:51:48.129501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.125621 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 19:51:48.137886 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.137845 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf"] Nov 02 19:51:48.138118 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.138092 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" podUID="7134c5c2-347a-4420-ba32-bd7d49741c59" containerName="maas-api" containerID="cri-o://2d38ab58692f35950a3042b6c571193ef3adf6308fea6ad60b6df3fe16675329" gracePeriod=30 Nov 02 19:51:48.643503 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.642486 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:48.955555 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.954764 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:51:48.970562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.968617 5850 generic.go:356] "Generic (PLEG): container finished" podID="7134c5c2-347a-4420-ba32-bd7d49741c59" containerID="2d38ab58692f35950a3042b6c571193ef3adf6308fea6ad60b6df3fe16675329" exitCode=0 Nov 02 19:51:48.970562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.968686 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" event={"ID":"7134c5c2-347a-4420-ba32-bd7d49741c59","Type":"ContainerDied","Data":"2d38ab58692f35950a3042b6c571193ef3adf6308fea6ad60b6df3fe16675329"} Nov 02 19:51:48.970562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.968734 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" event={"ID":"7134c5c2-347a-4420-ba32-bd7d49741c59","Type":"ContainerDied","Data":"eb4b19a42a53e82f400b5508cb5f25c0925485c4ca843f5ecc225e4895833fbf"} Nov 02 19:51:48.970562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.968749 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb4b19a42a53e82f400b5508cb5f25c0925485c4ca843f5ecc225e4895833fbf" Nov 02 19:51:48.970562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:48.970151 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:51:49.093499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.087029 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-metrics-certs\") pod \"7134c5c2-347a-4420-ba32-bd7d49741c59\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " Nov 02 19:51:49.093499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.087201 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7134c5c2-347a-4420-ba32-bd7d49741c59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7134c5c2-347a-4420-ba32-bd7d49741c59-openshift-service-ca\") pod \"7134c5c2-347a-4420-ba32-bd7d49741c59\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " Nov 02 19:51:49.093499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.087258 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls\") pod \"7134c5c2-347a-4420-ba32-bd7d49741c59\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " Nov 02 19:51:49.093499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.087554 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7134c5c2-347a-4420-ba32-bd7d49741c59-kube-api-access-bwmhn\" (UniqueName: \"kubernetes.io/projected/7134c5c2-347a-4420-ba32-bd7d49741c59-kube-api-access-bwmhn\") pod \"7134c5c2-347a-4420-ba32-bd7d49741c59\" (UID: \"7134c5c2-347a-4420-ba32-bd7d49741c59\") " Nov 02 19:51:49.129842 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.129800 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7134c5c2-347a-4420-ba32-bd7d49741c59-openshift-service-ca" pod "7134c5c2-347a-4420-ba32-bd7d49741c59" (UID: "7134c5c2-347a-4420-ba32-bd7d49741c59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:49.150691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.131643 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-metrics-certs" pod "7134c5c2-347a-4420-ba32-bd7d49741c59" (UID: "7134c5c2-347a-4420-ba32-bd7d49741c59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:51:49.150691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.132181 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls" pod "7134c5c2-347a-4420-ba32-bd7d49741c59" (UID: "7134c5c2-347a-4420-ba32-bd7d49741c59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:51:49.150691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.135108 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7134c5c2-347a-4420-ba32-bd7d49741c59-kube-api-access-bwmhn" pod "7134c5c2-347a-4420-ba32-bd7d49741c59" (UID: "7134c5c2-347a-4420-ba32-bd7d49741c59"). InnerVolumeSpecName "kube-api-access-bwmhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:49.228848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.208746 5850 patch_prober.go:29] interesting pod/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 19:51:49.228848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.208856 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 19:51:49.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.280195 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:49.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.280241 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7134c5c2-347a-4420-ba32-bd7d49741c59-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:49.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.280259 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7134c5c2-347a-4420-ba32-bd7d49741c59-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:49.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.280274 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwmhn\" (UniqueName: \"kubernetes.io/projected/7134c5c2-347a-4420-ba32-bd7d49741c59-kube-api-access-bwmhn\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:49.732439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.732387 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:49.951089 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.951054 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:49.976128 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:49.976097 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf" Nov 02 19:51:50.105550 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.101774 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:50.129500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.121818 5850 patch_prober.go:29] interesting pod/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 19:51:50.129500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.121891 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 19:51:50.193410 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.193365 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf"] Nov 02 19:51:50.197030 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.197000 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d23a1e27-64f999985d-zhjrf"] Nov 02 19:51:50.363558 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.362712 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn"] Nov 02 19:51:50.363558 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.363031 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" containerID="cri-o://e7bf97075ef6c8dd336c0a98f6fd5fb125575cb6f704f69ace35cd2cba4fb0e3" gracePeriod=30 Nov 02 19:51:50.431501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.430546 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:50.705500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.704699 5850 patch_prober.go:29] interesting pod/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:51:50.705500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:50.705089 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:51:51.056522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:51.056408 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:51.122777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:51.122738 5850 patch_prober.go:29] interesting pod/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 19:51:51.123061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:51.123032 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 19:51:51.852497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:51.852319 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7134c5c2-347a-4420-ba32-bd7d49741c59" path="/var/lib/kubelet/pods/7134c5c2-347a-4420-ba32-bd7d49741c59/volumes" Nov 02 19:51:52.123167 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:52.123124 5850 patch_prober.go:29] interesting pod/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 19:51:52.123356 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:52.123200 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 19:51:53.126113 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:53.126072 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:51:53.128323 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:53.128294 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:51:53.199328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:53.199291 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:53.408698 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:53.408607 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:54.739545 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:54.739465 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:56.023569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.023531 5850 generic.go:356] "Generic (PLEG): container finished" podID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerID="e7bf97075ef6c8dd336c0a98f6fd5fb125575cb6f704f69ace35cd2cba4fb0e3" exitCode=0 Nov 02 19:51:56.024110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.023767 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" event={"ID":"8a4ce348-3fa9-441f-ae13-40998c61314e","Type":"ContainerDied","Data":"e7bf97075ef6c8dd336c0a98f6fd5fb125575cb6f704f69ace35cd2cba4fb0e3"} Nov 02 19:51:56.024110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.023821 5850 scope.go:122] "RemoveContainer" containerID="a40db9bdfd22e9da82e6c99015f4f35ab11d9ff6cbd749761b1cf7a529e47584" Nov 02 19:51:56.070676 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.070653 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:51:56.165228 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165132 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-certs\") pod \"8a4ce348-3fa9-441f-ae13-40998c61314e\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " Nov 02 19:51:56.165228 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165181 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-credential-socket\") pod \"8a4ce348-3fa9-441f-ae13-40998c61314e\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " Nov 02 19:51:56.165525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165259 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-data\") pod \"8a4ce348-3fa9-441f-ae13-40998c61314e\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " Nov 02 19:51:56.165525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165289 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-socket\") pod \"8a4ce348-3fa9-441f-ae13-40998c61314e\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " Nov 02 19:51:56.165525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165333 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-token\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-token\") pod \"8a4ce348-3fa9-441f-ae13-40998c61314e\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " Nov 02 19:51:56.165525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165380 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-kube-api-access-jp2g7\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-kube-api-access-jp2g7\") pod \"8a4ce348-3fa9-441f-ae13-40998c61314e\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " Nov 02 19:51:56.165525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165453 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-envoy\") pod \"8a4ce348-3fa9-441f-ae13-40998c61314e\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " Nov 02 19:51:56.165795 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165508 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-certs" pod "8a4ce348-3fa9-441f-ae13-40998c61314e" (UID: "8a4ce348-3fa9-441f-ae13-40998c61314e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:51:56.165795 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165555 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4ce348-3fa9-441f-ae13-40998c61314e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4ce348-3fa9-441f-ae13-40998c61314e-istiod-ca-cert\") pod \"8a4ce348-3fa9-441f-ae13-40998c61314e\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " Nov 02 19:51:56.165795 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165593 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-credential-socket" pod "8a4ce348-3fa9-441f-ae13-40998c61314e" (UID: "8a4ce348-3fa9-441f-ae13-40998c61314e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:51:56.165795 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165640 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-data" pod "8a4ce348-3fa9-441f-ae13-40998c61314e" (UID: "8a4ce348-3fa9-441f-ae13-40998c61314e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:51:56.165795 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165685 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-socket" pod "8a4ce348-3fa9-441f-ae13-40998c61314e" (UID: "8a4ce348-3fa9-441f-ae13-40998c61314e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:51:56.165795 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165763 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-podinfo\") pod \"8a4ce348-3fa9-441f-ae13-40998c61314e\" (UID: \"8a4ce348-3fa9-441f-ae13-40998c61314e\") " Nov 02 19:51:56.166101 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.165915 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4ce348-3fa9-441f-ae13-40998c61314e-istiod-ca-cert" pod "8a4ce348-3fa9-441f-ae13-40998c61314e" (UID: "8a4ce348-3fa9-441f-ae13-40998c61314e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:56.166688 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.166662 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:56.166798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.166689 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:56.166798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.166705 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:56.166798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.166720 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:56.166798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.166736 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4ce348-3fa9-441f-ae13-40998c61314e-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:56.167811 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.167788 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-podinfo" pod "8a4ce348-3fa9-441f-ae13-40998c61314e" (UID: "8a4ce348-3fa9-441f-ae13-40998c61314e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:51:56.167906 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.167845 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-token" pod "8a4ce348-3fa9-441f-ae13-40998c61314e" (UID: "8a4ce348-3fa9-441f-ae13-40998c61314e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:56.167957 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.167907 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-kube-api-access-jp2g7" pod "8a4ce348-3fa9-441f-ae13-40998c61314e" (UID: "8a4ce348-3fa9-441f-ae13-40998c61314e"). InnerVolumeSpecName "kube-api-access-jp2g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:56.168045 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.168022 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-envoy" pod "8a4ce348-3fa9-441f-ae13-40998c61314e" (UID: "8a4ce348-3fa9-441f-ae13-40998c61314e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:51:56.268677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.268636 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:56.268677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.268680 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp2g7\" (UniqueName: \"kubernetes.io/projected/8a4ce348-3fa9-441f-ae13-40998c61314e-kube-api-access-jp2g7\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:56.268880 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.268697 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:56.268880 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:56.268713 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4ce348-3fa9-441f-ae13-40998c61314e-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:57.007464 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:57.007430 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:57.034546 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:57.034520 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" Nov 02 19:51:57.034993 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:57.034469 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn" event={"ID":"8a4ce348-3fa9-441f-ae13-40998c61314e","Type":"ContainerDied","Data":"12d42a99f1622965cecb6a05d83f29c9abcad516a828ae93a91490e324459153"} Nov 02 19:51:57.034993 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:57.034603 5850 scope.go:122] "RemoveContainer" containerID="e7bf97075ef6c8dd336c0a98f6fd5fb125575cb6f704f69ace35cd2cba4fb0e3" Nov 02 19:51:57.176466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:57.176430 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:57.336237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:57.336152 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn"] Nov 02 19:51:57.355844 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:57.355806 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d23a1e27-openshift-default-854dd4b679-p2npn"] Nov 02 19:51:57.447489 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:57.447450 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:57.816426 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:57.816390 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" path="/var/lib/kubelet/pods/8a4ce348-3fa9-441f-ae13-40998c61314e/volumes" Nov 02 19:51:58.774601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:58.774555 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84"] Nov 02 19:51:58.775025 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:58.774781 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" podUID="57c27b02-deed-4ed8-aed5-b6d134c071c7" containerName="maas-api" containerID="cri-o://c562096d922725304df696e6e7eb02ae0a538edcc185429d00ea48bcd139cc6e" gracePeriod=30 Nov 02 19:51:58.974256 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:58.974171 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:59.081813 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.081633 5850 generic.go:356] "Generic (PLEG): container finished" podID="57c27b02-deed-4ed8-aed5-b6d134c071c7" containerID="c562096d922725304df696e6e7eb02ae0a538edcc185429d00ea48bcd139cc6e" exitCode=0 Nov 02 19:51:59.082143 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.082072 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" event={"ID":"57c27b02-deed-4ed8-aed5-b6d134c071c7","Type":"ContainerDied","Data":"c562096d922725304df696e6e7eb02ae0a538edcc185429d00ea48bcd139cc6e"} Nov 02 19:51:59.374636 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.374605 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:51:59.529391 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.529355 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-metrics-certs\") pod \"57c27b02-deed-4ed8-aed5-b6d134c071c7\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " Nov 02 19:51:59.529572 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.529438 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c27b02-deed-4ed8-aed5-b6d134c071c7-kube-api-access-dj24l\" (UniqueName: \"kubernetes.io/projected/57c27b02-deed-4ed8-aed5-b6d134c071c7-kube-api-access-dj24l\") pod \"57c27b02-deed-4ed8-aed5-b6d134c071c7\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " Nov 02 19:51:59.529572 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.529539 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls\") pod \"57c27b02-deed-4ed8-aed5-b6d134c071c7\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " Nov 02 19:51:59.529666 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.529606 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c27b02-deed-4ed8-aed5-b6d134c071c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c27b02-deed-4ed8-aed5-b6d134c071c7-openshift-service-ca\") pod \"57c27b02-deed-4ed8-aed5-b6d134c071c7\" (UID: \"57c27b02-deed-4ed8-aed5-b6d134c071c7\") " Nov 02 19:51:59.530024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.529997 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c27b02-deed-4ed8-aed5-b6d134c071c7-openshift-service-ca" pod "57c27b02-deed-4ed8-aed5-b6d134c071c7" (UID: "57c27b02-deed-4ed8-aed5-b6d134c071c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:59.530287 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.530260 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c27b02-deed-4ed8-aed5-b6d134c071c7-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:59.532061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.532036 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-metrics-certs" pod "57c27b02-deed-4ed8-aed5-b6d134c071c7" (UID: "57c27b02-deed-4ed8-aed5-b6d134c071c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:51:59.532432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.532402 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls" pod "57c27b02-deed-4ed8-aed5-b6d134c071c7" (UID: "57c27b02-deed-4ed8-aed5-b6d134c071c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:51:59.532863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.532828 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c27b02-deed-4ed8-aed5-b6d134c071c7-kube-api-access-dj24l" pod "57c27b02-deed-4ed8-aed5-b6d134c071c7" (UID: "57c27b02-deed-4ed8-aed5-b6d134c071c7"). InnerVolumeSpecName "kube-api-access-dj24l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:59.631397 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.631316 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:59.631397 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.631346 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj24l\" (UniqueName: \"kubernetes.io/projected/57c27b02-deed-4ed8-aed5-b6d134c071c7-kube-api-access-dj24l\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:59.631397 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.631356 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c27b02-deed-4ed8-aed5-b6d134c071c7-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:59.832985 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.832949 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r"] Nov 02 19:51:59.833358 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.833186 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" podUID="5ce179b1-545b-497a-a471-2f18c4698b9a" containerName="maas-api" containerID="cri-o://25a4f5942cb220397a7cbabaace0a171b4595c9bb10ece6e90497ddd7275d504" gracePeriod=30 Nov 02 19:51:59.978989 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:51:59.978948 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:00.396265 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.396221 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" event={"ID":"57c27b02-deed-4ed8-aed5-b6d134c071c7","Type":"ContainerDied","Data":"6d97015e2616dcd32bde857cb8779be8f2e04be0c34ae03fa486baea3ba6a2d6"} Nov 02 19:52:00.396777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.396752 5850 scope.go:122] "RemoveContainer" containerID="c562096d922725304df696e6e7eb02ae0a538edcc185429d00ea48bcd139cc6e" Nov 02 19:52:00.397172 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.397153 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84" Nov 02 19:52:00.403566 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.403533 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:00.413171 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.413117 5850 generic.go:356] "Generic (PLEG): container finished" podID="5ce179b1-545b-497a-a471-2f18c4698b9a" containerID="25a4f5942cb220397a7cbabaace0a171b4595c9bb10ece6e90497ddd7275d504" exitCode=0 Nov 02 19:52:00.413624 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.413596 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" event={"ID":"5ce179b1-545b-497a-a471-2f18c4698b9a","Type":"ContainerDied","Data":"25a4f5942cb220397a7cbabaace0a171b4595c9bb10ece6e90497ddd7275d504"} Nov 02 19:52:00.548529 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.547151 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:52:00.572798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.572744 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:00.576608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.576578 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84"] Nov 02 19:52:00.579507 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.579462 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8ffab69-546485d64-cff84"] Nov 02 19:52:00.692535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.692447 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-metrics-certs\") pod \"5ce179b1-545b-497a-a471-2f18c4698b9a\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " Nov 02 19:52:00.692678 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.692545 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-maas-api-tls\") pod \"5ce179b1-545b-497a-a471-2f18c4698b9a\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " Nov 02 19:52:00.692678 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.692592 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce179b1-545b-497a-a471-2f18c4698b9a-kube-api-access-jm4jw\" (UniqueName: \"kubernetes.io/projected/5ce179b1-545b-497a-a471-2f18c4698b9a-kube-api-access-jm4jw\") pod \"5ce179b1-545b-497a-a471-2f18c4698b9a\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " Nov 02 19:52:00.692748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.692699 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce179b1-545b-497a-a471-2f18c4698b9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce179b1-545b-497a-a471-2f18c4698b9a-openshift-service-ca\") pod \"5ce179b1-545b-497a-a471-2f18c4698b9a\" (UID: \"5ce179b1-545b-497a-a471-2f18c4698b9a\") " Nov 02 19:52:00.693132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.693095 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce179b1-545b-497a-a471-2f18c4698b9a-openshift-service-ca" pod "5ce179b1-545b-497a-a471-2f18c4698b9a" (UID: "5ce179b1-545b-497a-a471-2f18c4698b9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:52:00.694267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.693691 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce179b1-545b-497a-a471-2f18c4698b9a-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:00.695082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.695053 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-maas-api-tls" pod "5ce179b1-545b-497a-a471-2f18c4698b9a" (UID: "5ce179b1-545b-497a-a471-2f18c4698b9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:52:00.695082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.695071 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce179b1-545b-497a-a471-2f18c4698b9a-kube-api-access-jm4jw" pod "5ce179b1-545b-497a-a471-2f18c4698b9a" (UID: "5ce179b1-545b-497a-a471-2f18c4698b9a"). InnerVolumeSpecName "kube-api-access-jm4jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:52:00.695253 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.695069 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-metrics-certs" pod "5ce179b1-545b-497a-a471-2f18c4698b9a" (UID: "5ce179b1-545b-497a-a471-2f18c4698b9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:52:00.757511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.756595 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:00.803327 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.797743 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:00.803665 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.803636 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce179b1-545b-497a-a471-2f18c4698b9a-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:00.803777 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.803764 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm4jw\" (UniqueName: \"kubernetes.io/projected/5ce179b1-545b-497a-a471-2f18c4698b9a-kube-api-access-jm4jw\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:00.836088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.836051 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z"] Nov 02 19:52:00.847002 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.846969 5850 cpu_manager.go:414] "containerMap: removing container" podUID="7134c5c2-347a-4420-ba32-bd7d49741c59" containerName="maas-api" Nov 02 19:52:00.847194 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.847181 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7134c5c2-347a-4420-ba32-bd7d49741c59" containerName="maas-api" Nov 02 19:52:00.847302 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.847292 5850 cpu_manager.go:414] "containerMap: removing container" podUID="5ce179b1-545b-497a-a471-2f18c4698b9a" containerName="maas-api" Nov 02 19:52:00.847382 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.847371 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce179b1-545b-497a-a471-2f18c4698b9a" containerName="maas-api" Nov 02 19:52:00.847492 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.847463 5850 cpu_manager.go:414] "containerMap: removing container" podUID="5d86b4de-f2e5-460d-b512-35694007416d" containerName="payload-processing" Nov 02 19:52:00.847602 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.847588 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d86b4de-f2e5-460d-b512-35694007416d" containerName="payload-processing" Nov 02 19:52:00.847719 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.847708 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" Nov 02 19:52:00.847791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.847781 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" Nov 02 19:52:00.847876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.847867 5850 cpu_manager.go:414] "containerMap: removing container" podUID="17112c9c-c35c-4466-b034-1a9ab330f5c5" containerName="payload-pre-processing" Nov 02 19:52:00.847954 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.847943 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17112c9c-c35c-4466-b034-1a9ab330f5c5" containerName="payload-pre-processing" Nov 02 19:52:00.848022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848012 5850 cpu_manager.go:414] "containerMap: removing container" podUID="5b5beb63-12ef-49f0-84b2-e02c617dccc7" containerName="payload-pre-processing" Nov 02 19:52:00.848096 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848086 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5beb63-12ef-49f0-84b2-e02c617dccc7" containerName="payload-pre-processing" Nov 02 19:52:00.848196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848187 5850 cpu_manager.go:414] "containerMap: removing container" podUID="57c27b02-deed-4ed8-aed5-b6d134c071c7" containerName="maas-api" Nov 02 19:52:00.848264 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848254 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c27b02-deed-4ed8-aed5-b6d134c071c7" containerName="maas-api" Nov 02 19:52:00.848513 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848498 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5beb63-12ef-49f0-84b2-e02c617dccc7" containerName="payload-pre-processing" Nov 02 19:52:00.848617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848608 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c27b02-deed-4ed8-aed5-b6d134c071c7" containerName="maas-api" Nov 02 19:52:00.848687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848679 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" Nov 02 19:52:00.848746 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848737 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="7134c5c2-347a-4420-ba32-bd7d49741c59" containerName="maas-api" Nov 02 19:52:00.848811 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848803 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce179b1-545b-497a-a471-2f18c4698b9a" containerName="maas-api" Nov 02 19:52:00.848878 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848870 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d86b4de-f2e5-460d-b512-35694007416d" containerName="payload-processing" Nov 02 19:52:00.848941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848933 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" Nov 02 19:52:00.849002 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.848994 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="17112c9c-c35c-4466-b034-1a9ab330f5c5" containerName="payload-pre-processing" Nov 02 19:52:00.867595 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.867563 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:00.933776 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.930643 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d368a053\"" Nov 02 19:52:01.001554 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:00.979754 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27"] Nov 02 19:52:01.235198 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.002834 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d368a053-dockercfg-8fnnr\"" Nov 02 19:52:01.235451 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.004870 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:01.235703 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.018261 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9c417f-18a1-4347-9f8b-136343404118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z\" (UID: \"ee9c417f-18a1-4347-9f8b-136343404118\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:01.236019 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.235999 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9lv2\" (UniqueName: \"kubernetes.io/projected/ee9c417f-18a1-4347-9f8b-136343404118-kube-api-access-x9lv2\") pod \"payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z\" (UID: \"ee9c417f-18a1-4347-9f8b-136343404118\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:01.389143 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.240781 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" Nov 02 19:52:01.389804 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.389775 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4ce348-3fa9-441f-ae13-40998c61314e" containerName="istio-proxy" Nov 02 19:52:01.495575 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.493236 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9lv2\" (UniqueName: \"kubernetes.io/projected/ee9c417f-18a1-4347-9f8b-136343404118-kube-api-access-x9lv2\") pod \"payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z\" (UID: \"ee9c417f-18a1-4347-9f8b-136343404118\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:01.495575 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.493621 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9c417f-18a1-4347-9f8b-136343404118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z\" (UID: \"ee9c417f-18a1-4347-9f8b-136343404118\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:01.495575 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.494599 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9c417f-18a1-4347-9f8b-136343404118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z\" (UID: \"ee9c417f-18a1-4347-9f8b-136343404118\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:01.519496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.496595 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:01.544331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.516046 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:01.547352 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.547315 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" event={"ID":"5ce179b1-545b-497a-a471-2f18c4698b9a","Type":"ContainerDied","Data":"63e1a56d07585dae0ca149e0fed674d3cc28f6a4bd257c5b333b3c71660a6827"} Nov 02 19:52:01.547601 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.547581 5850 scope.go:122] "RemoveContainer" containerID="25a4f5942cb220397a7cbabaace0a171b4595c9bb10ece6e90497ddd7275d504" Nov 02 19:52:01.549266 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.547642 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r" Nov 02 19:52:01.588210 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.588173 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27"] Nov 02 19:52:01.617502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.609004 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-plugins-config-volume\") pod \"payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27\" (UID: \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\") " pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:01.617502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.614727 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5x2\" (UniqueName: \"kubernetes.io/projected/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-kube-api-access-5w5x2\") pod \"payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27\" (UID: \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\") " pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:01.649249 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.648164 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9lv2\" (UniqueName: \"kubernetes.io/projected/ee9c417f-18a1-4347-9f8b-136343404118-kube-api-access-x9lv2\") pod \"payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z\" (UID: \"ee9c417f-18a1-4347-9f8b-136343404118\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:01.665645 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.665611 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z"] Nov 02 19:52:01.715887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.715855 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-plugins-config-volume\") pod \"payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27\" (UID: \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\") " pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:01.715887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.715888 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5x2\" (UniqueName: \"kubernetes.io/projected/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-kube-api-access-5w5x2\") pod \"payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27\" (UID: \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\") " pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:01.716587 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.716567 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-plugins-config-volume\") pod \"payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27\" (UID: \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\") " pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:01.747723 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.747690 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:01.775407 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.775368 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5x2\" (UniqueName: \"kubernetes.io/projected/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-kube-api-access-5w5x2\") pod \"payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27\" (UID: \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\") " pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:01.818241 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.818099 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c27b02-deed-4ed8-aed5-b6d134c071c7" path="/var/lib/kubelet/pods/57c27b02-deed-4ed8-aed5-b6d134c071c7/volumes" Nov 02 19:52:01.830781 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.830746 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r"] Nov 02 19:52:01.836682 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.836646 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d4e9ef2-6cb88748d6-bn57r"] Nov 02 19:52:01.881661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.881630 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d368a053-dockercfg-8fnnr\"" Nov 02 19:52:01.889375 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.889348 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:01.913073 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.913024 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:01.974585 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:01.968276 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:02.108579 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.096772 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:02.470950 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:52:02.470915 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9c417f_18a1_4347_9f8b_136343404118.slice/crio-23616c8e731b6a5363814a2293edec00a20f4966014831070f9c95e2d9c1f9ee WatchSource:0}: Error finding container 23616c8e731b6a5363814a2293edec00a20f4966014831070f9c95e2d9c1f9ee: Status 404 returned error can't find the container with id 23616c8e731b6a5363814a2293edec00a20f4966014831070f9c95e2d9c1f9ee Nov 02 19:52:02.474074 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.474045 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z"] Nov 02 19:52:02.526257 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.526228 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b"] Nov 02 19:52:02.526516 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.526460 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" containerID="cri-o://b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6" gracePeriod=30 Nov 02 19:52:02.548164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.548116 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:02.557845 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.557803 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" event={"ID":"ee9c417f-18a1-4347-9f8b-136343404118","Type":"ContainerStarted","Data":"8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e"} Nov 02 19:52:02.558007 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.557871 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" event={"ID":"ee9c417f-18a1-4347-9f8b-136343404118","Type":"ContainerStarted","Data":"23616c8e731b6a5363814a2293edec00a20f4966014831070f9c95e2d9c1f9ee"} Nov 02 19:52:02.558007 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.557900 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:02.846531 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.837590 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27"] Nov 02 19:52:02.874535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:02.868688 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" podStartSLOduration=2.868668564 podStartE2EDuration="2.868668564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:02.792159968 +0000 UTC m=+1381.134549050" watchObservedRunningTime="2026-11-02 19:52:02.868668564 +0000 UTC m=+1381.211057645" Nov 02 19:52:03.198635 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.198600 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:03.566391 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.566352 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" event={"ID":"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52","Type":"ContainerStarted","Data":"d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994"} Nov 02 19:52:03.566618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.566412 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" event={"ID":"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52","Type":"ContainerStarted","Data":"1468bff0a9e13498e2e191d634e13cb8a7d738efac1bcf9790ff60226155ab6f"} Nov 02 19:52:03.566618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.566449 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:03.680240 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.680200 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz"] Nov 02 19:52:03.684707 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.684682 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.685986 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.685937 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" podStartSLOduration=3.685924799 podStartE2EDuration="3.685924799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:03.682829703 +0000 UTC m=+1382.025218783" watchObservedRunningTime="2026-11-02 19:52:03.685924799 +0000 UTC m=+1382.028313879" Nov 02 19:52:03.740896 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.740860 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-46f86d84-openshift-default-dockercfg-pc9zv\"" Nov 02 19:52:03.786821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.786784 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz"] Nov 02 19:52:03.795839 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.795810 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:03.817320 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.817228 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce179b1-545b-497a-a471-2f18c4698b9a" path="/var/lib/kubelet/pods/5ce179b1-545b-497a-a471-2f18c4698b9a/volumes" Nov 02 19:52:03.841590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.841543 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-token\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.841816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.841639 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-certs\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.841816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.841723 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istiod-ca-cert\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.841816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.841785 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-credential-socket\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.841816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.841811 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbcnx\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-kube-api-access-wbcnx\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.883500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.877989 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-data\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.883500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.878073 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-envoy\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.883500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.878182 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-socket\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.883500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.878267 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-podinfo\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:03.991501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:03.990424 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:04.013630 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.005889 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-certs\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.015953 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-certs\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.016109 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istiod-ca-cert\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.016195 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-credential-socket\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.016224 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbcnx\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-kube-api-access-wbcnx\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.016387 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-data\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.016429 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-envoy\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.016537 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-socket\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.016620 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-podinfo\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.016687 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-token\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.017889 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istiod-ca-cert\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.018213 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-credential-socket\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.022506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.018813 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-data\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.023240 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.023069 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-socket\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.070239 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.069215 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-podinfo\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.070239 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.069527 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-envoy\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.144499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.137990 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-token\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.153558 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.148827 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbcnx\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-kube-api-access-wbcnx\") pod \"e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.302422 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.301453 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:04.734208 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:52:04.734174 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab6ece9_b656_4a56_ad6e_1a3299cb239b.slice/crio-89ded070c07fdc98747663b8ed1f6b142b5408a4657967c91a529d38b2fb1a2c WatchSource:0}: Error finding container 89ded070c07fdc98747663b8ed1f6b142b5408a4657967c91a529d38b2fb1a2c: Status 404 returned error can't find the container with id 89ded070c07fdc98747663b8ed1f6b142b5408a4657967c91a529d38b2fb1a2c Nov 02 19:52:04.736109 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.736084 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz"] Nov 02 19:52:04.737039 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.737002 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:04.737137 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.737085 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:04.737203 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:04.737131 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:05.580732 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:05.580696 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" event={"ID":"8ab6ece9-b656-4a56-ad6e-1a3299cb239b","Type":"ContainerStarted","Data":"547194ff78096e8eaf61179ad25e071976443fdff5bee0c2001d7758813fc4c7"} Nov 02 19:52:05.581139 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:05.580746 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" event={"ID":"8ab6ece9-b656-4a56-ad6e-1a3299cb239b","Type":"ContainerStarted","Data":"89ded070c07fdc98747663b8ed1f6b142b5408a4657967c91a529d38b2fb1a2c"} Nov 02 19:52:05.803452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:05.803388 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" podStartSLOduration=2.803370619 podStartE2EDuration="2.803370619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:05.751131047 +0000 UTC m=+1384.093520126" watchObservedRunningTime="2026-11-02 19:52:05.803370619 +0000 UTC m=+1384.145759701" Nov 02 19:52:06.302162 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:06.302124 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:06.308051 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:06.308021 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:06.588174 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:06.588082 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:06.588993 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:06.588974 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:52:06.836561 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:06.836522 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:06.996065 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:06.993968 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:07.796246 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:07.796177 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:07.905906 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:07.905877 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:52:07.937873 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:07.937840 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6"] Nov 02 19:52:07.938514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:07.938494 5850 cpu_manager.go:414] "containerMap: removing container" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" Nov 02 19:52:07.938514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:07.938514 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" Nov 02 19:52:07.938660 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:07.938541 5850 cpu_manager.go:414] "containerMap: removing container" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" Nov 02 19:52:07.938660 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:07.938546 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" Nov 02 19:52:07.938721 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:07.938665 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" Nov 02 19:52:07.942213 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:07.942192 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.001590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.001554 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-4848d2-openshift-default-dockercfg-bfwcj\"" Nov 02 19:52:08.026461 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.026429 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:08.032755 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.032721 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-socket\") pod \"950d65f4-05cf-48ed-a7f5-570a3c11a675\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " Nov 02 19:52:08.032916 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.032799 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-podinfo\") pod \"950d65f4-05cf-48ed-a7f5-570a3c11a675\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " Nov 02 19:52:08.032916 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.032884 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-certs\") pod \"950d65f4-05cf-48ed-a7f5-570a3c11a675\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " Nov 02 19:52:08.032916 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.032912 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950d65f4-05cf-48ed-a7f5-570a3c11a675-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d65f4-05cf-48ed-a7f5-570a3c11a675-istiod-ca-cert\") pod \"950d65f4-05cf-48ed-a7f5-570a3c11a675\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " Nov 02 19:52:08.033069 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.032966 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-socket" pod "950d65f4-05cf-48ed-a7f5-570a3c11a675" (UID: "950d65f4-05cf-48ed-a7f5-570a3c11a675"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:52:08.033069 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.033012 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-envoy\") pod \"950d65f4-05cf-48ed-a7f5-570a3c11a675\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " Nov 02 19:52:08.033176 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.033094 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-token\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-token\") pod \"950d65f4-05cf-48ed-a7f5-570a3c11a675\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " Nov 02 19:52:08.033277 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.033254 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-certs" pod "950d65f4-05cf-48ed-a7f5-570a3c11a675" (UID: "950d65f4-05cf-48ed-a7f5-570a3c11a675"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:52:08.033356 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.033263 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-data\") pod \"950d65f4-05cf-48ed-a7f5-570a3c11a675\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " Nov 02 19:52:08.033356 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.033337 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-credential-socket\") pod \"950d65f4-05cf-48ed-a7f5-570a3c11a675\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " Nov 02 19:52:08.033455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.033366 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d65f4-05cf-48ed-a7f5-570a3c11a675-istiod-ca-cert" pod "950d65f4-05cf-48ed-a7f5-570a3c11a675" (UID: "950d65f4-05cf-48ed-a7f5-570a3c11a675"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:52:08.033534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.033456 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-kube-api-access-ll4vw\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-kube-api-access-ll4vw\") pod \"950d65f4-05cf-48ed-a7f5-570a3c11a675\" (UID: \"950d65f4-05cf-48ed-a7f5-570a3c11a675\") " Nov 02 19:52:08.033534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.033489 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-data" pod "950d65f4-05cf-48ed-a7f5-570a3c11a675" (UID: "950d65f4-05cf-48ed-a7f5-570a3c11a675"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:52:08.033646 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.033623 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-credential-socket" pod "950d65f4-05cf-48ed-a7f5-570a3c11a675" (UID: "950d65f4-05cf-48ed-a7f5-570a3c11a675"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:52:08.034213 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.034185 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:08.034308 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.034217 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:08.034308 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.034233 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:08.034308 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.034246 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:08.034308 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.034259 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d65f4-05cf-48ed-a7f5-570a3c11a675-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:08.036298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.036256 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-envoy" pod "950d65f4-05cf-48ed-a7f5-570a3c11a675" (UID: "950d65f4-05cf-48ed-a7f5-570a3c11a675"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:52:08.036298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.036271 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-podinfo" pod "950d65f4-05cf-48ed-a7f5-570a3c11a675" (UID: "950d65f4-05cf-48ed-a7f5-570a3c11a675"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:52:08.036463 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.036301 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-token" pod "950d65f4-05cf-48ed-a7f5-570a3c11a675" (UID: "950d65f4-05cf-48ed-a7f5-570a3c11a675"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:52:08.036626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.036603 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-kube-api-access-ll4vw" pod "950d65f4-05cf-48ed-a7f5-570a3c11a675" (UID: "950d65f4-05cf-48ed-a7f5-570a3c11a675"). InnerVolumeSpecName "kube-api-access-ll4vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:52:08.043142 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.043118 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6"] Nov 02 19:52:08.150089 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.149993 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kcr\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-kube-api-access-r4kcr\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.150367 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.150343 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-certs\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.150504 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.150487 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-credential-socket\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.150795 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.150770 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-data\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.150931 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.150915 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1564eb-9776-44b3-9a46-310ff4d64841-istiod-ca-cert\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.151132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.151113 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-podinfo\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.151460 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.151440 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-envoy\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.155501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.151779 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-token\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.155723 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.155699 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-socket\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.171497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.169347 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll4vw\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-kube-api-access-ll4vw\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:08.171497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.169388 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:08.171497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.169405 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:08.171497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.169418 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d65f4-05cf-48ed-a7f5-570a3c11a675-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:52:08.219502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.215993 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:08.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.282334 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-token\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.282415 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-socket\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.282525 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kcr\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-kube-api-access-r4kcr\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.282611 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-certs\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.282639 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-credential-socket\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.282731 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-data\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.282906 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1564eb-9776-44b3-9a46-310ff4d64841-istiod-ca-cert\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.282990 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-podinfo\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.284496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.283041 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-envoy\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.298316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.298262 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-socket\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.306610 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.306563 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-envoy\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.308438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.308406 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-data\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.331631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.320525 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1564eb-9776-44b3-9a46-310ff4d64841-istiod-ca-cert\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.331833 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.323719 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-certs\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.332891 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.323958 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-credential-socket\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.361120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.361009 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-podinfo\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.363321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.363291 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-token\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.390456 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.390385 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kcr\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-kube-api-access-r4kcr\") pod \"e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.535573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.535346 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:08.562995 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.561682 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:08.639322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.639284 5850 generic.go:356] "Generic (PLEG): container finished" podID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerID="b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6" exitCode=0 Nov 02 19:52:08.639322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.639321 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" event={"ID":"950d65f4-05cf-48ed-a7f5-570a3c11a675","Type":"ContainerDied","Data":"b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6"} Nov 02 19:52:08.639694 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.639363 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" Nov 02 19:52:08.639694 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.639388 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b" event={"ID":"950d65f4-05cf-48ed-a7f5-570a3c11a675","Type":"ContainerDied","Data":"07faf471400c8818fe8e25a75cb6bd931489b9a213804fdfec56a4a827581f91"} Nov 02 19:52:08.639694 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.639404 5850 scope.go:122] "RemoveContainer" containerID="b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6" Nov 02 19:52:08.651653 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.651622 5850 scope.go:122] "RemoveContainer" containerID="e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954" Nov 02 19:52:08.668351 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.668180 5850 scope.go:122] "RemoveContainer" containerID="b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6" Nov 02 19:52:08.668615 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:52:08.668595 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6\": container with ID starting with b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6 not found: ID does not exist" containerID="b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6" Nov 02 19:52:08.668663 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.668625 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6"} err="failed to get container status \"b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6\": rpc error: code = NotFound desc = could not find container \"b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6\": container with ID starting with b48dd636a1adcc5d75b3eedac86f08bc2d359a07a65191a791735b895f3d5fc6 not found: ID does not exist" Nov 02 19:52:08.668663 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.668647 5850 scope.go:122] "RemoveContainer" containerID="e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954" Nov 02 19:52:08.668955 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:52:08.668928 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954\": container with ID starting with e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954 not found: ID does not exist" containerID="e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954" Nov 02 19:52:08.669051 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.668965 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954"} err="failed to get container status \"e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954\": rpc error: code = NotFound desc = could not find container \"e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954\": container with ID starting with e4c68b4ed0d51c21b3fe9c444b01c46b0c98f94da24f3a06d56b0424efb20954 not found: ID does not exist" Nov 02 19:52:08.812053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.811949 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b"] Nov 02 19:52:08.815281 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:08.815256 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6d4e9ef2-openshift-default-68f77df667-22j2b"] Nov 02 19:52:09.047723 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:52:09.047685 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1564eb_9776_44b3_9a46_310ff4d64841.slice/crio-11108a9bfd7a453866a69fbc8940627f9ba0a9e50096ad734236a6545c620498 WatchSource:0}: Error finding container 11108a9bfd7a453866a69fbc8940627f9ba0a9e50096ad734236a6545c620498: Status 404 returned error can't find the container with id 11108a9bfd7a453866a69fbc8940627f9ba0a9e50096ad734236a6545c620498 Nov 02 19:52:09.049785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:09.049756 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6"] Nov 02 19:52:09.050540 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:09.050506 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:09.050643 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:09.050578 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:09.050643 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:09.050608 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:09.648843 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:09.648796 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" event={"ID":"4e1564eb-9776-44b3-9a46-310ff4d64841","Type":"ContainerStarted","Data":"13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd"} Nov 02 19:52:09.648843 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:09.648847 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" event={"ID":"4e1564eb-9776-44b3-9a46-310ff4d64841","Type":"ContainerStarted","Data":"11108a9bfd7a453866a69fbc8940627f9ba0a9e50096ad734236a6545c620498"} Nov 02 19:52:09.773426 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:09.773370 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" podStartSLOduration=2.773355121 podStartE2EDuration="2.773355121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:09.770327787 +0000 UTC m=+1388.112716867" watchObservedRunningTime="2026-11-02 19:52:09.773355121 +0000 UTC m=+1388.115744198" Nov 02 19:52:09.816395 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:09.816360 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" path="/var/lib/kubelet/pods/950d65f4-05cf-48ed-a7f5-570a3c11a675/volumes" Nov 02 19:52:10.562683 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:10.562644 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:10.568428 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:10.568400 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:10.655908 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:10.655880 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:10.656966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:10.656948 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:52:11.138395 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.138362 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:11.333525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.329591 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:11.511306 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.511175 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:11.673363 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.673321 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9"] Nov 02 19:52:11.679514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.676397 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="950d65f4-05cf-48ed-a7f5-570a3c11a675" containerName="istio-proxy" Nov 02 19:52:11.705132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.703881 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:11.754826 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.754789 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:11.772437 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.764550 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9"] Nov 02 19:52:11.851500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.847784 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-46f86d84\"" Nov 02 19:52:11.851500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:11.848006 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-46f86d84-dockercfg-cdcbm\"" Nov 02 19:52:12.026792 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.026691 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:12.045493 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.038519 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2989c9-8271-45bd-91a5-e54e008e14de-plugins-config-volume\") pod \"payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9\" (UID: \"af2989c9-8271-45bd-91a5-e54e008e14de\") " pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:12.045493 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.038742 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f9lh\" (UniqueName: \"kubernetes.io/projected/af2989c9-8271-45bd-91a5-e54e008e14de-kube-api-access-8f9lh\") pod \"payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9\" (UID: \"af2989c9-8271-45bd-91a5-e54e008e14de\") " pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:12.140130 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.140097 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f9lh\" (UniqueName: \"kubernetes.io/projected/af2989c9-8271-45bd-91a5-e54e008e14de-kube-api-access-8f9lh\") pod \"payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9\" (UID: \"af2989c9-8271-45bd-91a5-e54e008e14de\") " pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:12.140608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.140274 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2989c9-8271-45bd-91a5-e54e008e14de-plugins-config-volume\") pod \"payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9\" (UID: \"af2989c9-8271-45bd-91a5-e54e008e14de\") " pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:12.140997 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.140977 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2989c9-8271-45bd-91a5-e54e008e14de-plugins-config-volume\") pod \"payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9\" (UID: \"af2989c9-8271-45bd-91a5-e54e008e14de\") " pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:12.198379 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.198344 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f9lh\" (UniqueName: \"kubernetes.io/projected/af2989c9-8271-45bd-91a5-e54e008e14de-kube-api-access-8f9lh\") pod \"payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9\" (UID: \"af2989c9-8271-45bd-91a5-e54e008e14de\") " pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:12.417653 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.417613 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:12.580853 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.580822 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:12.793811 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:52:12.793774 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2989c9_8271_45bd_91a5_e54e008e14de.slice/crio-2f70b160a9e9083d6db31f180eedde67989a8ce40096e9f2c40e0345f25ee3e8 WatchSource:0}: Error finding container 2f70b160a9e9083d6db31f180eedde67989a8ce40096e9f2c40e0345f25ee3e8: Status 404 returned error can't find the container with id 2f70b160a9e9083d6db31f180eedde67989a8ce40096e9f2c40e0345f25ee3e8 Nov 02 19:52:12.796086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.796046 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:12.797329 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:12.797307 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9"] Nov 02 19:52:13.621205 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:13.621173 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:52:13.690917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:13.690881 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" event={"ID":"af2989c9-8271-45bd-91a5-e54e008e14de","Type":"ContainerStarted","Data":"ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad"} Nov 02 19:52:13.691108 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:13.690941 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" event={"ID":"af2989c9-8271-45bd-91a5-e54e008e14de","Type":"ContainerStarted","Data":"2f70b160a9e9083d6db31f180eedde67989a8ce40096e9f2c40e0345f25ee3e8"} Nov 02 19:52:13.691108 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:13.691051 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:14.577148 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:14.577120 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:52:14.695613 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:14.695555 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" podStartSLOduration=3.695542364 podStartE2EDuration="3.695542364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:13.874049749 +0000 UTC m=+1392.216438856" watchObservedRunningTime="2026-11-02 19:52:14.695542364 +0000 UTC m=+1393.037931452" Nov 02 19:52:14.765291 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:52:14.765249 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:52:16.416408 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:16.416353 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:16.765195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:16.765106 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:16.948542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:16.948500 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk"] Nov 02 19:52:16.960748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:16.960714 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:16.965788 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:16.965758 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:17.080553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.064984 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-4848d2\"" Nov 02 19:52:17.080553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.065157 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-4848d2-dockercfg-7k9s9\"" Nov 02 19:52:17.081667 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.081626 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80659f4-faea-4edc-81e9-e525fed8afeb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk\" (UID: \"e80659f4-faea-4edc-81e9-e525fed8afeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:17.108371 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.108331 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmpp\" (UniqueName: \"kubernetes.io/projected/e80659f4-faea-4edc-81e9-e525fed8afeb-kube-api-access-ghmpp\") pod \"payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk\" (UID: \"e80659f4-faea-4edc-81e9-e525fed8afeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:17.108963 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.103898 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk"] Nov 02 19:52:17.319920 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.319878 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649"] Nov 02 19:52:17.338280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.322298 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80659f4-faea-4edc-81e9-e525fed8afeb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk\" (UID: \"e80659f4-faea-4edc-81e9-e525fed8afeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:17.361499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.323193 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80659f4-faea-4edc-81e9-e525fed8afeb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk\" (UID: \"e80659f4-faea-4edc-81e9-e525fed8afeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:17.361499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.338900 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmpp\" (UniqueName: \"kubernetes.io/projected/e80659f4-faea-4edc-81e9-e525fed8afeb-kube-api-access-ghmpp\") pod \"payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk\" (UID: \"e80659f4-faea-4edc-81e9-e525fed8afeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:17.361499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.352038 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649"] Nov 02 19:52:17.361499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.352160 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:17.415530 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.414044 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmpp\" (UniqueName: \"kubernetes.io/projected/e80659f4-faea-4edc-81e9-e525fed8afeb-kube-api-access-ghmpp\") pod \"payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk\" (UID: \"e80659f4-faea-4edc-81e9-e525fed8afeb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:17.441101 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.441063 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b596-bf27-4ec8-b864-8eb365f150fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649\" (UID: \"6e76b596-bf27-4ec8-b864-8eb365f150fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:17.441516 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.441218 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsv8k\" (UniqueName: \"kubernetes.io/projected/6e76b596-bf27-4ec8-b864-8eb365f150fc-kube-api-access-qsv8k\") pod \"payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649\" (UID: \"6e76b596-bf27-4ec8-b864-8eb365f150fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:17.542374 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.542333 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b596-bf27-4ec8-b864-8eb365f150fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649\" (UID: \"6e76b596-bf27-4ec8-b864-8eb365f150fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:17.542563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.542418 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsv8k\" (UniqueName: \"kubernetes.io/projected/6e76b596-bf27-4ec8-b864-8eb365f150fc-kube-api-access-qsv8k\") pod \"payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649\" (UID: \"6e76b596-bf27-4ec8-b864-8eb365f150fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:17.543059 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.543038 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b596-bf27-4ec8-b864-8eb365f150fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649\" (UID: \"6e76b596-bf27-4ec8-b864-8eb365f150fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:17.598217 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.598129 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:17.603439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.603416 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsv8k\" (UniqueName: \"kubernetes.io/projected/6e76b596-bf27-4ec8-b864-8eb365f150fc-kube-api-access-qsv8k\") pod \"payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649\" (UID: \"6e76b596-bf27-4ec8-b864-8eb365f150fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:17.676834 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.676547 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:17.973778 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:52:17.973742 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80659f4_faea_4edc_81e9_e525fed8afeb.slice/crio-d8fc316ea3851987222c9b808227ff0c470312190daa4d80533e37748eba335e WatchSource:0}: Error finding container d8fc316ea3851987222c9b808227ff0c470312190daa4d80533e37748eba335e: Status 404 returned error can't find the container with id d8fc316ea3851987222c9b808227ff0c470312190daa4d80533e37748eba335e Nov 02 19:52:17.976345 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:17.976315 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk"] Nov 02 19:52:18.063812 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:52:18.063776 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e76b596_bf27_4ec8_b864_8eb365f150fc.slice/crio-83d0544ae49cf5d418eb790f50bca1982de6eed4cbe3a92e36ee19bfc41cd397 WatchSource:0}: Error finding container 83d0544ae49cf5d418eb790f50bca1982de6eed4cbe3a92e36ee19bfc41cd397: Status 404 returned error can't find the container with id 83d0544ae49cf5d418eb790f50bca1982de6eed4cbe3a92e36ee19bfc41cd397 Nov 02 19:52:18.067022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:18.067001 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649"] Nov 02 19:52:18.802257 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:18.802226 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" event={"ID":"6e76b596-bf27-4ec8-b864-8eb365f150fc","Type":"ContainerStarted","Data":"a8f064fe844b287a9bdfd040cb3060f6a2f90315cfab660310c0af90d1426c14"} Nov 02 19:52:18.802784 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:18.802277 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" event={"ID":"6e76b596-bf27-4ec8-b864-8eb365f150fc","Type":"ContainerStarted","Data":"83d0544ae49cf5d418eb790f50bca1982de6eed4cbe3a92e36ee19bfc41cd397"} Nov 02 19:52:18.802784 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:18.802428 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:18.804301 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:18.804276 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" event={"ID":"e80659f4-faea-4edc-81e9-e525fed8afeb","Type":"ContainerStarted","Data":"d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789"} Nov 02 19:52:18.804966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:18.804319 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" event={"ID":"e80659f4-faea-4edc-81e9-e525fed8afeb","Type":"ContainerStarted","Data":"d8fc316ea3851987222c9b808227ff0c470312190daa4d80533e37748eba335e"} Nov 02 19:52:18.804966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:18.804357 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:18.920797 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:18.920738 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" podStartSLOduration=1.9207234359999998 podStartE2EDuration="1.920723436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:18.919691908 +0000 UTC m=+1397.262080988" watchObservedRunningTime="2026-11-02 19:52:18.920723436 +0000 UTC m=+1397.263112515" Nov 02 19:52:21.443998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:21.443962 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:24.708551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:24.708519 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:52:24.825455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:24.825400 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" podStartSLOduration=8.825387451 podStartE2EDuration="8.825387451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:19.043550583 +0000 UTC m=+1397.385939662" watchObservedRunningTime="2026-11-02 19:52:24.825387451 +0000 UTC m=+1403.167776542" Nov 02 19:52:25.938417 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:25.938380 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:29.816587 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:29.816556 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 19:52:29.817027 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:29.816622 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 19:52:30.347443 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:30.347405 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:34.750126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:34.750081 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:34.943876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:34.943836 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q"] Nov 02 19:52:34.949415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:34.949389 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:34.995629 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:34.995587 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j"] Nov 02 19:52:35.000085 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.000059 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.046862 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.046825 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6gd\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-kube-api-access-hm6gd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.046862 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.046874 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.047125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.046900 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.047125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047030 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.047125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047069 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.047125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047089 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.047291 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047163 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.047291 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047187 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.047291 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047209 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.047291 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047281 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.047452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047304 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.047452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047364 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1ee935-9e4f-4e64-8e87-7821569346cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.047452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047383 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.047452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047437 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.047640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047557 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.047640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047608 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwsv7\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-kube-api-access-nwsv7\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.047640 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047633 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.047778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.047659 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.053106 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.053077 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q"] Nov 02 19:52:35.106719 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.106685 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j"] Nov 02 19:52:35.149096 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149054 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6gd\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-kube-api-access-hm6gd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.149316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149108 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.149316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149141 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.149316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149174 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.149316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149214 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.149316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149237 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.149316 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149285 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.149687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149320 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.149687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149352 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.149687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149436 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.149687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149494 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.149687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149573 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1ee935-9e4f-4e64-8e87-7821569346cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.149687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149603 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.149687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149631 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.150038 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149695 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.150038 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149776 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwsv7\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-kube-api-access-nwsv7\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.150038 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149812 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.150038 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.149844 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.151214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.150698 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.151214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.150765 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.151214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.150904 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.151214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.151088 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.151581 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.151262 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.151650 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.151612 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.152920 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.152098 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.152920 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.152309 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.152920 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.152777 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1ee935-9e4f-4e64-8e87-7821569346cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.153197 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.153166 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.153444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.153425 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.155044 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.154765 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.155044 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.154952 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.155318 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.155274 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.209536 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.209467 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6gd\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-kube-api-access-hm6gd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.209942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.209919 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jrr5q\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.210048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.210024 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.210710 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.210690 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwsv7\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-kube-api-access-nwsv7\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2xg2j\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.266737 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.266639 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:35.315696 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.315656 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:35.695453 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:52:35.695416 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dc06fe_3a71_49a9_9cdd_98a54700ca96.slice/crio-968bed8fa9f7c4adfe9aef52149ca6ff5d174979de2dca563a040da89a175e67 WatchSource:0}: Error finding container 968bed8fa9f7c4adfe9aef52149ca6ff5d174979de2dca563a040da89a175e67: Status 404 returned error can't find the container with id 968bed8fa9f7c4adfe9aef52149ca6ff5d174979de2dca563a040da89a175e67 Nov 02 19:52:35.698408 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.698369 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:35.698536 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.698457 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:35.698536 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.698523 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:35.698681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.698658 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q"] Nov 02 19:52:35.747430 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:52:35.747400 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1ee935_9e4f_4e64_8e87_7821569346cc.slice/crio-56e71f342a2ecc78660e48e9d242e83671065e6f6ec41118afe5868a9c4275d0 WatchSource:0}: Error finding container 56e71f342a2ecc78660e48e9d242e83671065e6f6ec41118afe5868a9c4275d0: Status 404 returned error can't find the container with id 56e71f342a2ecc78660e48e9d242e83671065e6f6ec41118afe5868a9c4275d0 Nov 02 19:52:35.749907 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.749880 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j"] Nov 02 19:52:35.750275 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.750245 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:35.750704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.750318 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:35.750704 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.750365 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:52:35.924195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.924157 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" event={"ID":"f3dc06fe-3a71-49a9-9cdd-98a54700ca96","Type":"ContainerStarted","Data":"99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e"} Nov 02 19:52:35.924692 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.924224 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" event={"ID":"f3dc06fe-3a71-49a9-9cdd-98a54700ca96","Type":"ContainerStarted","Data":"968bed8fa9f7c4adfe9aef52149ca6ff5d174979de2dca563a040da89a175e67"} Nov 02 19:52:35.925930 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.925904 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" event={"ID":"2c1ee935-9e4f-4e64-8e87-7821569346cc","Type":"ContainerStarted","Data":"e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8"} Nov 02 19:52:35.926062 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:35.925948 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" event={"ID":"2c1ee935-9e4f-4e64-8e87-7821569346cc","Type":"ContainerStarted","Data":"56e71f342a2ecc78660e48e9d242e83671065e6f6ec41118afe5868a9c4275d0"} Nov 02 19:52:36.045645 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:36.045524 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podStartSLOduration=2.04550751 podStartE2EDuration="2.04550751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:36.042235592 +0000 UTC m=+1414.384624673" watchObservedRunningTime="2026-11-02 19:52:36.04550751 +0000 UTC m=+1414.387896591" Nov 02 19:52:36.162363 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:36.162289 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podStartSLOduration=2.162263028 podStartE2EDuration="2.162263028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:36.161105658 +0000 UTC m=+1414.503494738" watchObservedRunningTime="2026-11-02 19:52:36.162263028 +0000 UTC m=+1414.504652111" Nov 02 19:52:36.269497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:36.268797 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:36.270671 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:36.270635 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 19:52:36.270835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:36.270707 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 19:52:36.320138 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:36.320038 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:36.323398 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:36.323357 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 19:52:36.323643 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:36.323428 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 19:52:37.268052 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:37.268011 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 19:52:37.268531 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:37.268069 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 19:52:37.327500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:37.316971 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 19:52:37.327500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:37.317044 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 19:52:38.267919 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:38.267883 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 19:52:38.268117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:38.267939 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 19:52:38.331505 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:38.324853 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 19:52:38.331505 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:38.324920 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 19:52:38.886442 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:38.886406 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57"] Nov 02 19:52:38.892552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:38.892526 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:38.966542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:38.963714 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57"] Nov 02 19:52:38.997411 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:38.997381 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d368a053-serving-cert\"" Nov 02 19:52:39.057162 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.057121 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2c5\" (UniqueName: \"kubernetes.io/projected/55986bdf-983a-4ecb-a0a4-fd785a06066d-kube-api-access-kv2c5\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.057671 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.057636 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-metrics-certs\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.058509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.058247 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55986bdf-983a-4ecb-a0a4-fd785a06066d-openshift-service-ca\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.058822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.058803 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-maas-api-tls\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.091587 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.091501 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:39.207557 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.204622 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55986bdf-983a-4ecb-a0a4-fd785a06066d-openshift-service-ca\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.207557 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.204709 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-maas-api-tls\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.207557 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.204990 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2c5\" (UniqueName: \"kubernetes.io/projected/55986bdf-983a-4ecb-a0a4-fd785a06066d-kube-api-access-kv2c5\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.207557 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.205017 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-metrics-certs\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.222590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.222136 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55986bdf-983a-4ecb-a0a4-fd785a06066d-openshift-service-ca\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.245068 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.244991 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:39.259518 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.249547 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-metrics-certs\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.275761 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.275201 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 19:52:39.275761 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.275296 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 19:52:39.286496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.280064 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-maas-api-tls\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.318681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.312663 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2c5\" (UniqueName: \"kubernetes.io/projected/55986bdf-983a-4ecb-a0a4-fd785a06066d-kube-api-access-kv2c5\") pod \"maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:39.318681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.318545 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 19:52:39.318681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.318612 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 19:52:39.434397 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.434342 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:39.563552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:39.533950 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:40.070057 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:52:40.070017 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55986bdf_983a_4ecb_a0a4_fd785a06066d.slice/crio-6513c23c56cad6c03ff0919f1347df208b0e775f64379d6c23b40fcd1eacc301 WatchSource:0}: Error finding container 6513c23c56cad6c03ff0919f1347df208b0e775f64379d6c23b40fcd1eacc301: Status 404 returned error can't find the container with id 6513c23c56cad6c03ff0919f1347df208b0e775f64379d6c23b40fcd1eacc301 Nov 02 19:52:40.073120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:40.073097 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57"] Nov 02 19:52:40.267372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:40.267332 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 19:52:40.267552 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:40.267397 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 19:52:40.331435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:40.331339 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 19:52:40.332094 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:40.332059 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 19:52:41.097466 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:41.097416 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" event={"ID":"55986bdf-983a-4ecb-a0a4-fd785a06066d","Type":"ContainerStarted","Data":"6513c23c56cad6c03ff0919f1347df208b0e775f64379d6c23b40fcd1eacc301"} Nov 02 19:52:41.285199 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:41.285167 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 19:52:41.285948 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:41.285544 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 19:52:41.319032 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:41.318990 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 19:52:41.319295 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:41.319260 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 19:52:42.173822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:42.173782 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" event={"ID":"55986bdf-983a-4ecb-a0a4-fd785a06066d","Type":"ContainerStarted","Data":"be6ffcf4eeea40c9a386a334380bc7d3239e3c3424d06d784ce12a448bced1a7"} Nov 02 19:52:42.175329 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:42.175304 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:42.267095 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:42.267057 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 19:52:42.267451 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:42.267118 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 19:52:42.296263 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:42.296190 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" podStartSLOduration=3.082490306 podStartE2EDuration="4.29616739s" totalImagesPullingTime="1.213677084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:42.291742774 +0000 UTC m=+1420.634131858" watchObservedRunningTime="2026-11-02 19:52:42.29616739 +0000 UTC m=+1420.638556471" Nov 02 19:52:42.316173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:42.316134 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 19:52:42.316324 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:42.316198 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 19:52:43.268051 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:43.268005 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 19:52:43.268455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:43.268081 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 19:52:43.316846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:43.316801 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 19:52:43.317028 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:43.316877 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 19:52:43.672868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:43.672834 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:44.789352 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:52:44.789305 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:52:45.002830 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:45.002800 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:45.003390 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:45.003362 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:45.004189 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:45.004152 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 19:52:45.053101 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:45.053004 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:45.054042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:45.054009 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:45.055048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:45.055030 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 19:52:48.105059 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:48.105032 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:48.296517 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:48.296454 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:48.451605 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:48.451573 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:49.201922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:49.201892 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:52:50.928028 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:50.927902 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:51.588425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:51.588372 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:52.761996 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:52.761949 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:57.305689 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:57.305646 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:57.465417 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:57.463170 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:57.635454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:52:57.635373 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:02.033269 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:02.033227 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:06.518285 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:06.518213 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:10.943920 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:10.943824 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:12.245768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:12.245424 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:13.453468 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:13.453432 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:14.333529 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:14.333461 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:14.762836 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:53:14.762784 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:53:15.025571 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:15.023674 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:15.965998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:15.965948 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:16.148366 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:16.148184 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:16.334760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:16.334669 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:18.676636 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:18.676597 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:18.831412 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:18.831376 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:19.452986 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:19.452950 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:19.633159 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:19.633126 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:20.001563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:19.993816 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:20.190000 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.179181 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:20.413571 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.413530 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585"] Nov 02 19:53:20.473832 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.473799 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7"] Nov 02 19:53:20.474349 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.474318 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" containerID="cri-o://e5df1ab4987429c70c1150b58cb9b77f07708659bcf9184c24bec086e4a9e4c9" gracePeriod=30 Nov 02 19:53:20.475216 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.474115 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:20.533063 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.532886 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4099cbd3\"" Nov 02 19:53:20.533063 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.533028 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4099cbd3-dockercfg-s5rk6\"" Nov 02 19:53:20.534631 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.534602 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585"] Nov 02 19:53:20.663494 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.562204 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68zs\" (UniqueName: \"kubernetes.io/projected/6444026e-0d2d-4cc7-923e-7013345cb966-kube-api-access-z68zs\") pod \"payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585\" (UID: \"6444026e-0d2d-4cc7-923e-7013345cb966\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:20.724258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.724167 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6444026e-0d2d-4cc7-923e-7013345cb966-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585\" (UID: \"6444026e-0d2d-4cc7-923e-7013345cb966\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:20.887511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.880225 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68zs\" (UniqueName: \"kubernetes.io/projected/6444026e-0d2d-4cc7-923e-7013345cb966-kube-api-access-z68zs\") pod \"payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585\" (UID: \"6444026e-0d2d-4cc7-923e-7013345cb966\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:20.887511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.880347 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6444026e-0d2d-4cc7-923e-7013345cb966-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585\" (UID: \"6444026e-0d2d-4cc7-923e-7013345cb966\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:20.924195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.922740 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6444026e-0d2d-4cc7-923e-7013345cb966-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585\" (UID: \"6444026e-0d2d-4cc7-923e-7013345cb966\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:20.938774 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:20.938733 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:53:21.017555 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:21.010293 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68zs\" (UniqueName: \"kubernetes.io/projected/6444026e-0d2d-4cc7-923e-7013345cb966-kube-api-access-z68zs\") pod \"payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585\" (UID: \"6444026e-0d2d-4cc7-923e-7013345cb966\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:21.090378 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:21.090201 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:21.286886 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:21.286780 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:21.776004 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:53:21.775970 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6444026e_0d2d_4cc7_923e_7013345cb966.slice/crio-7c50288bceb139e9e6f21a9253dbbfe47eb98ad23fc0c1fcfef555bc98916113 WatchSource:0}: Error finding container 7c50288bceb139e9e6f21a9253dbbfe47eb98ad23fc0c1fcfef555bc98916113: Status 404 returned error can't find the container with id 7c50288bceb139e9e6f21a9253dbbfe47eb98ad23fc0c1fcfef555bc98916113 Nov 02 19:53:21.779962 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:21.779936 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585"] Nov 02 19:53:22.157870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:22.157768 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" event={"ID":"6444026e-0d2d-4cc7-923e-7013345cb966","Type":"ContainerStarted","Data":"ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78"} Nov 02 19:53:22.157870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:22.157837 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" event={"ID":"6444026e-0d2d-4cc7-923e-7013345cb966","Type":"ContainerStarted","Data":"7c50288bceb139e9e6f21a9253dbbfe47eb98ad23fc0c1fcfef555bc98916113"} Nov 02 19:53:22.159003 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:22.158980 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:22.281331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:22.281259 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" podStartSLOduration=2.281240209 podStartE2EDuration="2.281240209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:22.278161082 +0000 UTC m=+1460.620550162" watchObservedRunningTime="2026-11-02 19:53:22.281240209 +0000 UTC m=+1460.623629290" Nov 02 19:53:22.630675 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:22.630626 5850 patch_prober.go:29] interesting pod/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:53:22.630823 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:22.630709 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:53:25.316542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:25.316509 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:25.598715 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:25.591531 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:25.677494 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:25.668908 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 19:53:25.677494 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:25.668989 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 19:53:25.767111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:25.765112 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:26.126520 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.119876 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2"] Nov 02 19:53:26.285591 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.285504 5850 generic.go:356] "Generic (PLEG): container finished" podID="c753243f-3ba8-46c1-97d3-517915d494f3" containerID="e5df1ab4987429c70c1150b58cb9b77f07708659bcf9184c24bec086e4a9e4c9" exitCode=0 Nov 02 19:53:26.343600 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.343561 5850 generic.go:356] "Generic (PLEG): container finished" podID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerID="13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd" exitCode=0 Nov 02 19:53:26.344637 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.344462 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:26.348331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.344363 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" event={"ID":"c753243f-3ba8-46c1-97d3-517915d494f3","Type":"ContainerDied","Data":"e5df1ab4987429c70c1150b58cb9b77f07708659bcf9184c24bec086e4a9e4c9"} Nov 02 19:53:26.348331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.346708 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" event={"ID":"4e1564eb-9776-44b3-9a46-310ff4d64841","Type":"ContainerDied","Data":"13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd"} Nov 02 19:53:26.348331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.346748 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2"] Nov 02 19:53:26.348331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.347283 5850 scope.go:122] "RemoveContainer" containerID="13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd" Nov 02 19:53:26.352034 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.351884 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:53:26.353955 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.352262 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:53:26.353955 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.352331 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:53:26.406439 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.406398 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-4fd0d5\"" Nov 02 19:53:26.451617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.451575 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-4fd0d5-dockercfg-dc9zd\"" Nov 02 19:53:26.486061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.486027 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:53:26.515005 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.514964 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237eb2b7-58b5-4acc-97c2-084473f92f0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2\" (UID: \"237eb2b7-58b5-4acc-97c2-084473f92f0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:26.515549 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.515269 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7qb\" (UniqueName: \"kubernetes.io/projected/237eb2b7-58b5-4acc-97c2-084473f92f0c-kube-api-access-cf7qb\") pod \"payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2\" (UID: \"237eb2b7-58b5-4acc-97c2-084473f92f0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.616394 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c753243f-3ba8-46c1-97d3-517915d494f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c753243f-3ba8-46c1-97d3-517915d494f3-istiod-ca-cert\") pod \"c753243f-3ba8-46c1-97d3-517915d494f3\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.616466 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-certs\") pod \"c753243f-3ba8-46c1-97d3-517915d494f3\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.616540 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-istio-token\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-istio-token\") pod \"c753243f-3ba8-46c1-97d3-517915d494f3\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.616785 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-envoy\") pod \"c753243f-3ba8-46c1-97d3-517915d494f3\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.616843 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-kube-api-access-h72hb\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-kube-api-access-h72hb\") pod \"c753243f-3ba8-46c1-97d3-517915d494f3\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.616889 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c753243f-3ba8-46c1-97d3-517915d494f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c753243f-3ba8-46c1-97d3-517915d494f3-istio-podinfo\") pod \"c753243f-3ba8-46c1-97d3-517915d494f3\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.616943 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-credential-socket\") pod \"c753243f-3ba8-46c1-97d3-517915d494f3\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.617066 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-socket\") pod \"c753243f-3ba8-46c1-97d3-517915d494f3\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.617109 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-data\") pod \"c753243f-3ba8-46c1-97d3-517915d494f3\" (UID: \"c753243f-3ba8-46c1-97d3-517915d494f3\") " Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.617566 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237eb2b7-58b5-4acc-97c2-084473f92f0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2\" (UID: \"237eb2b7-58b5-4acc-97c2-084473f92f0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:26.618618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.617834 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7qb\" (UniqueName: \"kubernetes.io/projected/237eb2b7-58b5-4acc-97c2-084473f92f0c-kube-api-access-cf7qb\") pod \"payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2\" (UID: \"237eb2b7-58b5-4acc-97c2-084473f92f0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:26.629130 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.628893 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-kube-api-access-h72hb" pod "c753243f-3ba8-46c1-97d3-517915d494f3" (UID: "c753243f-3ba8-46c1-97d3-517915d494f3"). InnerVolumeSpecName "kube-api-access-h72hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:26.629733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.629434 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c753243f-3ba8-46c1-97d3-517915d494f3-istiod-ca-cert" pod "c753243f-3ba8-46c1-97d3-517915d494f3" (UID: "c753243f-3ba8-46c1-97d3-517915d494f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:26.633518 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.633266 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c753243f-3ba8-46c1-97d3-517915d494f3-istio-podinfo" pod "c753243f-3ba8-46c1-97d3-517915d494f3" (UID: "c753243f-3ba8-46c1-97d3-517915d494f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:53:26.633518 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.633302 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-envoy" pod "c753243f-3ba8-46c1-97d3-517915d494f3" (UID: "c753243f-3ba8-46c1-97d3-517915d494f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.633690 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.633550 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-certs" pod "c753243f-3ba8-46c1-97d3-517915d494f3" (UID: "c753243f-3ba8-46c1-97d3-517915d494f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.633900 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.633751 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-data" pod "c753243f-3ba8-46c1-97d3-517915d494f3" (UID: "c753243f-3ba8-46c1-97d3-517915d494f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.633900 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.633789 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-credential-socket" pod "c753243f-3ba8-46c1-97d3-517915d494f3" (UID: "c753243f-3ba8-46c1-97d3-517915d494f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.634034 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.633944 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-socket" pod "c753243f-3ba8-46c1-97d3-517915d494f3" (UID: "c753243f-3ba8-46c1-97d3-517915d494f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.634845 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.634789 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237eb2b7-58b5-4acc-97c2-084473f92f0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2\" (UID: \"237eb2b7-58b5-4acc-97c2-084473f92f0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:26.635510 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.635449 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-istio-token" pod "c753243f-3ba8-46c1-97d3-517915d494f3" (UID: "c753243f-3ba8-46c1-97d3-517915d494f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:26.692245 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.692216 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7qb\" (UniqueName: \"kubernetes.io/projected/237eb2b7-58b5-4acc-97c2-084473f92f0c-kube-api-access-cf7qb\") pod \"payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2\" (UID: \"237eb2b7-58b5-4acc-97c2-084473f92f0c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:26.729727 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.729690 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.729727 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.729728 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h72hb\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-kube-api-access-h72hb\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.729944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.729743 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c753243f-3ba8-46c1-97d3-517915d494f3-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.729944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.729756 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.729944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.729769 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.729944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.729780 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.729944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.729792 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c753243f-3ba8-46c1-97d3-517915d494f3-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.729944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.729803 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c753243f-3ba8-46c1-97d3-517915d494f3-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.729944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.729827 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c753243f-3ba8-46c1-97d3-517915d494f3-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.989282 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:26.989178 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:27.353110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.353020 5850 generic.go:356] "Generic (PLEG): container finished" podID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerID="547194ff78096e8eaf61179ad25e071976443fdff5bee0c2001d7758813fc4c7" exitCode=0 Nov 02 19:53:27.353110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.353091 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" event={"ID":"8ab6ece9-b656-4a56-ad6e-1a3299cb239b","Type":"ContainerDied","Data":"547194ff78096e8eaf61179ad25e071976443fdff5bee0c2001d7758813fc4c7"} Nov 02 19:53:27.353710 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.353690 5850 scope.go:122] "RemoveContainer" containerID="547194ff78096e8eaf61179ad25e071976443fdff5bee0c2001d7758813fc4c7" Nov 02 19:53:27.355369 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.355346 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" event={"ID":"4e1564eb-9776-44b3-9a46-310ff4d64841","Type":"ContainerStarted","Data":"2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262"} Nov 02 19:53:27.355675 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.355641 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:53:27.355765 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.355730 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:53:27.355820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.355775 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:53:27.356864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.356844 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" event={"ID":"c753243f-3ba8-46c1-97d3-517915d494f3","Type":"ContainerDied","Data":"9033e8b4651a1da7385c56586cac79b4f077fc473681aebe97a40642adf761fe"} Nov 02 19:53:27.356968 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.356888 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7" Nov 02 19:53:27.356968 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.356897 5850 scope.go:122] "RemoveContainer" containerID="e5df1ab4987429c70c1150b58cb9b77f07708659bcf9184c24bec086e4a9e4c9" Nov 02 19:53:27.378706 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.378668 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2"] Nov 02 19:53:27.384210 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:53:27.384172 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237eb2b7_58b5_4acc_97c2_084473f92f0c.slice/crio-0568e7c6c9823335079301d6b7b1bd6fd053b99179823c883e585e93aec562bc WatchSource:0}: Error finding container 0568e7c6c9823335079301d6b7b1bd6fd053b99179823c883e585e93aec562bc: Status 404 returned error can't find the container with id 0568e7c6c9823335079301d6b7b1bd6fd053b99179823c883e585e93aec562bc Nov 02 19:53:27.564088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.564049 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:53:27.564448 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.564412 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 19:53:27.564660 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.564627 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 19:53:27.670435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.670283 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7"] Nov 02 19:53:27.674294 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.674224 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-c8mp7"] Nov 02 19:53:27.842358 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:27.842323 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" path="/var/lib/kubelet/pods/c753243f-3ba8-46c1-97d3-517915d494f3/volumes" Nov 02 19:53:28.368587 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:28.368552 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" event={"ID":"237eb2b7-58b5-4acc-97c2-084473f92f0c","Type":"ContainerStarted","Data":"7d283c4398ad4330b28d76a7dd8aab51776bddd06c302afc0a1a865e68dbb69e"} Nov 02 19:53:28.369110 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:28.369094 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:28.369216 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:28.369200 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" event={"ID":"237eb2b7-58b5-4acc-97c2-084473f92f0c","Type":"ContainerStarted","Data":"0568e7c6c9823335079301d6b7b1bd6fd053b99179823c883e585e93aec562bc"} Nov 02 19:53:28.370808 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:28.370788 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" event={"ID":"8ab6ece9-b656-4a56-ad6e-1a3299cb239b","Type":"ContainerStarted","Data":"a6f9c228f414c01b519129da8e20b378998a76043c719fb23ff616a1b77efa2a"} Nov 02 19:53:28.540524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:28.528687 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" podStartSLOduration=3.528667623 podStartE2EDuration="3.528667623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:28.503367108 +0000 UTC m=+1466.845756188" watchObservedRunningTime="2026-11-02 19:53:28.528667623 +0000 UTC m=+1466.871056703" Nov 02 19:53:28.563195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:28.563155 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 19:53:28.563378 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:28.563222 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 19:53:29.302668 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:29.302629 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:53:29.302885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:29.302859 5850 patch_prober.go:29] interesting pod/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 19:53:29.302994 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:29.302920 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 19:53:29.562849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:29.562542 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 19:53:29.562849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:29.562627 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 19:53:30.302459 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:30.302412 5850 patch_prober.go:29] interesting pod/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 19:53:30.302685 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:30.302507 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 19:53:30.363821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:30.363787 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:30.564128 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:30.564027 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 19:53:30.564776 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:30.564726 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 19:53:31.302988 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:31.302949 5850 patch_prober.go:29] interesting pod/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 19:53:31.303202 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:31.303020 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 19:53:31.562682 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:31.562565 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 19:53:31.562682 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:31.562639 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 19:53:32.302377 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:32.302335 5850 patch_prober.go:29] interesting pod/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 19:53:32.302974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:32.302393 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 19:53:32.563435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:32.563351 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 19:53:32.563435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:32.563405 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 19:53:33.302363 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:33.302327 5850 patch_prober.go:29] interesting pod/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 19:53:33.302591 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:33.302388 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 19:53:33.567661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:33.567572 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:53:33.567906 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:33.567883 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:53:33.568737 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:33.568717 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 19:53:34.226499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:34.214707 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:53:34.316838 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:34.316801 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:53:34.322611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:34.320601 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:53:34.324569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:34.324542 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:53:35.062459 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:35.062422 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:39.382965 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:39.382933 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 19:53:39.638312 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:39.638220 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:44.563537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:44.563502 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:44.766515 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:53:44.766440 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:53:49.153293 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:49.153259 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:51.322882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:51.322846 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27"] Nov 02 19:53:51.447462 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:51.447422 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z"] Nov 02 19:53:51.736903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:51.736870 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:51.915290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:51.914026 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:54.144817 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:54.144779 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:54.628085 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:54.628048 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:55.085933 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.085851 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:55.557184 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.557146 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52"] Nov 02 19:53:55.557940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.557920 5850 cpu_manager.go:414] "containerMap: removing container" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" Nov 02 19:53:55.557940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.557937 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" Nov 02 19:53:55.558082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.558041 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="c753243f-3ba8-46c1-97d3-517915d494f3" containerName="istio-proxy" Nov 02 19:53:55.561037 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.561019 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.609797 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.609758 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-credential-socket\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.610116 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.610095 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-podinfo\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.610229 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.610214 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-data\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.610363 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.610350 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-socket\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.610598 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.610555 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-envoy\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.610882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.610856 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84rc\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-kube-api-access-w84rc\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.611124 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.611105 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8dc527-83b5-48cd-8726-efe1912a67c9-istiod-ca-cert\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.611257 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.611239 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-token\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.611526 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.611507 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-certs\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.621415 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.621384 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-282875-openshift-default-dockercfg-x6892\"" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.772729 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-certs\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.772948 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-credential-socket\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.773015 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-podinfo\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.773078 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-data\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.773158 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-socket\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.773199 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-envoy\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.773248 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84rc\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-kube-api-access-w84rc\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.773375 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8dc527-83b5-48cd-8726-efe1912a67c9-istiod-ca-cert\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.773408 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-token\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.780355 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-data\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.780683 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-certs\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.780940 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-credential-socket\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.824243 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.798419 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-socket\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.824243 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.813273 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:55.824243 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.816343 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8dc527-83b5-48cd-8726-efe1912a67c9-istiod-ca-cert\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.896524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.893802 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-podinfo\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.937531 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.936240 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-envoy\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:55.983192 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:55.982720 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84rc\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-kube-api-access-w84rc\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:56.014576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.012683 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-token\") pod \"e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:56.056835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.056790 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52"] Nov 02 19:53:56.120767 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.120716 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:56.176266 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.175831 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:56.323842 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.323782 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" podUID="07280bf0-3a94-4bd6-b9d7-ae55ea44fe52" containerName="payload-processing" containerID="cri-o://d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994" gracePeriod=25 Nov 02 19:53:56.447840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.447793 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" podUID="ee9c417f-18a1-4347-9f8b-136343404118" containerName="payload-pre-processing" containerID="cri-o://8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e" gracePeriod=25 Nov 02 19:53:56.602444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.602413 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:53:56.644896 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.644861 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-plugins-config-volume\") pod \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\" (UID: \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\") " Nov 02 19:53:56.645097 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.644917 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-kube-api-access-5w5x2\" (UniqueName: \"kubernetes.io/projected/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-kube-api-access-5w5x2\") pod \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\" (UID: \"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52\") " Nov 02 19:53:56.645302 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.645265 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-plugins-config-volume" pod "07280bf0-3a94-4bd6-b9d7-ae55ea44fe52" (UID: "07280bf0-3a94-4bd6-b9d7-ae55ea44fe52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:56.645934 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.645842 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:56.647314 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.647291 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-kube-api-access-5w5x2" pod "07280bf0-3a94-4bd6-b9d7-ae55ea44fe52" (UID: "07280bf0-3a94-4bd6-b9d7-ae55ea44fe52"). InnerVolumeSpecName "kube-api-access-5w5x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:56.666879 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.666840 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52"] Nov 02 19:53:56.698909 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:53:56.698872 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8dc527_83b5_48cd_8726_efe1912a67c9.slice/crio-dac5120c5569de0dea4124276b27016a0f0af0d6d455e47c39e3efd8be0fc952 WatchSource:0}: Error finding container dac5120c5569de0dea4124276b27016a0f0af0d6d455e47c39e3efd8be0fc952: Status 404 returned error can't find the container with id dac5120c5569de0dea4124276b27016a0f0af0d6d455e47c39e3efd8be0fc952 Nov 02 19:53:56.702377 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.702335 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:53:56.702465 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.702439 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:53:56.702694 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.702525 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:53:56.747229 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.747197 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5x2\" (UniqueName: \"kubernetes.io/projected/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52-kube-api-access-5w5x2\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:56.766149 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.766119 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:53:56.798331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.798285 5850 generic.go:356] "Generic (PLEG): container finished" podID="07280bf0-3a94-4bd6-b9d7-ae55ea44fe52" containerID="d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994" exitCode=0 Nov 02 19:53:56.798510 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.798354 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" Nov 02 19:53:56.798510 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.798365 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" event={"ID":"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52","Type":"ContainerDied","Data":"d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994"} Nov 02 19:53:56.798510 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.798420 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27" event={"ID":"07280bf0-3a94-4bd6-b9d7-ae55ea44fe52","Type":"ContainerDied","Data":"1468bff0a9e13498e2e191d634e13cb8a7d738efac1bcf9790ff60226155ab6f"} Nov 02 19:53:56.798510 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.798451 5850 scope.go:122] "RemoveContainer" containerID="d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994" Nov 02 19:53:56.800557 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.800532 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" event={"ID":"ed8dc527-83b5-48cd-8726-efe1912a67c9","Type":"ContainerStarted","Data":"5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b"} Nov 02 19:53:56.800675 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.800577 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" event={"ID":"ed8dc527-83b5-48cd-8726-efe1912a67c9","Type":"ContainerStarted","Data":"dac5120c5569de0dea4124276b27016a0f0af0d6d455e47c39e3efd8be0fc952"} Nov 02 19:53:56.802807 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.802784 5850 generic.go:356] "Generic (PLEG): container finished" podID="ee9c417f-18a1-4347-9f8b-136343404118" containerID="8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e" exitCode=0 Nov 02 19:53:56.802900 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.802814 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" event={"ID":"ee9c417f-18a1-4347-9f8b-136343404118","Type":"ContainerDied","Data":"8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e"} Nov 02 19:53:56.802900 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.802841 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" event={"ID":"ee9c417f-18a1-4347-9f8b-136343404118","Type":"ContainerDied","Data":"23616c8e731b6a5363814a2293edec00a20f4966014831070f9c95e2d9c1f9ee"} Nov 02 19:53:56.802900 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.802883 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z" Nov 02 19:53:56.813267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.813245 5850 scope.go:122] "RemoveContainer" containerID="d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994" Nov 02 19:53:56.813694 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:53:56.813666 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994\": container with ID starting with d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994 not found: ID does not exist" containerID="d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994" Nov 02 19:53:56.813791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.813708 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994"} err="failed to get container status \"d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994\": rpc error: code = NotFound desc = could not find container \"d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994\": container with ID starting with d7a40647f57d145231c1eec4395674a740bfd1b7a60db341afccd2c498e5c994 not found: ID does not exist" Nov 02 19:53:56.813791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.813734 5850 scope.go:122] "RemoveContainer" containerID="8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e" Nov 02 19:53:56.827379 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.827352 5850 scope.go:122] "RemoveContainer" containerID="8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e" Nov 02 19:53:56.827835 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:53:56.827769 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e\": container with ID starting with 8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e not found: ID does not exist" containerID="8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e" Nov 02 19:53:56.827835 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.827807 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e"} err="failed to get container status \"8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e\": rpc error: code = NotFound desc = could not find container \"8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e\": container with ID starting with 8f22e7c19c3b1943ab63bb5edf6b6661701a46d2f95173fe81ae54d88eab2c0e not found: ID does not exist" Nov 02 19:53:56.950237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.950127 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9c417f-18a1-4347-9f8b-136343404118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9c417f-18a1-4347-9f8b-136343404118-plugins-config-volume\") pod \"ee9c417f-18a1-4347-9f8b-136343404118\" (UID: \"ee9c417f-18a1-4347-9f8b-136343404118\") " Nov 02 19:53:56.950237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.950230 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9c417f-18a1-4347-9f8b-136343404118-kube-api-access-x9lv2\" (UniqueName: \"kubernetes.io/projected/ee9c417f-18a1-4347-9f8b-136343404118-kube-api-access-x9lv2\") pod \"ee9c417f-18a1-4347-9f8b-136343404118\" (UID: \"ee9c417f-18a1-4347-9f8b-136343404118\") " Nov 02 19:53:56.950619 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.950592 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9c417f-18a1-4347-9f8b-136343404118-plugins-config-volume" pod "ee9c417f-18a1-4347-9f8b-136343404118" (UID: "ee9c417f-18a1-4347-9f8b-136343404118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:56.951076 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.951051 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9c417f-18a1-4347-9f8b-136343404118-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:56.952710 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:56.952684 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9c417f-18a1-4347-9f8b-136343404118-kube-api-access-x9lv2" pod "ee9c417f-18a1-4347-9f8b-136343404118" (UID: "ee9c417f-18a1-4347-9f8b-136343404118"). InnerVolumeSpecName "kube-api-access-x9lv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:57.011629 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.011568 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" podStartSLOduration=2.011550123 podStartE2EDuration="2.011550123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:57.010379249 +0000 UTC m=+1495.352768366" watchObservedRunningTime="2026-11-02 19:53:57.011550123 +0000 UTC m=+1495.353939203" Nov 02 19:53:57.052309 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.052271 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9lv2\" (UniqueName: \"kubernetes.io/projected/ee9c417f-18a1-4347-9f8b-136343404118-kube-api-access-x9lv2\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:57.185204 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.173413 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27"] Nov 02 19:53:57.185204 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.178815 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:57.197701 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.187724 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d368a053-748dd7f58c-r6d27"] Nov 02 19:53:57.198003 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.190411 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 19:53:57.198268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.198242 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 19:53:57.314706 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.314618 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z"] Nov 02 19:53:57.318445 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.318416 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d368a053-5ff6f88967-mbp6z"] Nov 02 19:53:57.817299 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.817269 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07280bf0-3a94-4bd6-b9d7-ae55ea44fe52" path="/var/lib/kubelet/pods/07280bf0-3a94-4bd6-b9d7-ae55ea44fe52/volumes" Nov 02 19:53:57.817865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:57.817845 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9c417f-18a1-4347-9f8b-136343404118" path="/var/lib/kubelet/pods/ee9c417f-18a1-4347-9f8b-136343404118/volumes" Nov 02 19:53:58.181956 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:58.181921 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:58.819377 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:58.819348 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:58.820627 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:58.820605 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:53:59.216388 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:59.216357 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:59.775970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:59.775921 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:59.941592 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:53:59.941551 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:00.111027 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:00.110914 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:01.625639 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:01.625605 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9"] Nov 02 19:54:02.024117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:02.023922 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:02.256829 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:02.241940 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:04.040068 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:04.040000 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:05.766735 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:05.766689 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:06.078548 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.070951 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:06.184768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.183061 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq"] Nov 02 19:54:06.184768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.183847 5850 cpu_manager.go:414] "containerMap: removing container" podUID="ee9c417f-18a1-4347-9f8b-136343404118" containerName="payload-pre-processing" Nov 02 19:54:06.184768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.183866 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9c417f-18a1-4347-9f8b-136343404118" containerName="payload-pre-processing" Nov 02 19:54:06.184768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.183887 5850 cpu_manager.go:414] "containerMap: removing container" podUID="07280bf0-3a94-4bd6-b9d7-ae55ea44fe52" containerName="payload-processing" Nov 02 19:54:06.184768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.183892 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07280bf0-3a94-4bd6-b9d7-ae55ea44fe52" containerName="payload-processing" Nov 02 19:54:06.184768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.184044 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9c417f-18a1-4347-9f8b-136343404118" containerName="payload-pre-processing" Nov 02 19:54:06.184768 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.184059 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="07280bf0-3a94-4bd6-b9d7-ae55ea44fe52" containerName="payload-processing" Nov 02 19:54:06.188251 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.188224 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:06.229858 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.229817 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq"] Nov 02 19:54:06.250581 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.250533 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba308b4-8fec-4518-a5c3-245ed4f543f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq\" (UID: \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:06.250930 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.250898 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jds2\" (UniqueName: \"kubernetes.io/projected/9ba308b4-8fec-4518-a5c3-245ed4f543f7-kube-api-access-7jds2\") pod \"payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq\" (UID: \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:06.281267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.279784 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-282875-dockercfg-dcdfs\"" Nov 02 19:54:06.281267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.279864 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-282875\"" Nov 02 19:54:06.376553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.360211 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba308b4-8fec-4518-a5c3-245ed4f543f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq\" (UID: \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:06.376553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.360441 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jds2\" (UniqueName: \"kubernetes.io/projected/9ba308b4-8fec-4518-a5c3-245ed4f543f7-kube-api-access-7jds2\") pod \"payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq\" (UID: \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:06.430495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.402458 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba308b4-8fec-4518-a5c3-245ed4f543f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq\" (UID: \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:06.430951 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.430902 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7"] Nov 02 19:54:06.490372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.490345 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:06.632701 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.632588 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" podUID="af2989c9-8271-45bd-91a5-e54e008e14de" containerName="payload-processing" containerID="cri-o://ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad" gracePeriod=25 Nov 02 19:54:06.660392 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.660363 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jds2\" (UniqueName: \"kubernetes.io/projected/9ba308b4-8fec-4518-a5c3-245ed4f543f7-kube-api-access-7jds2\") pod \"payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq\" (UID: \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:06.665517 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.664036 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7"] Nov 02 19:54:06.745569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.745533 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcpjp\" (UniqueName: \"kubernetes.io/projected/365d84dc-cf64-4d85-9d5b-05647ab37fd1-kube-api-access-xcpjp\") pod \"payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7\" (UID: \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:06.745817 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.745790 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365d84dc-cf64-4d85-9d5b-05647ab37fd1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7\" (UID: \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:06.802494 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.802441 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:06.847557 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.847494 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcpjp\" (UniqueName: \"kubernetes.io/projected/365d84dc-cf64-4d85-9d5b-05647ab37fd1-kube-api-access-xcpjp\") pod \"payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7\" (UID: \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:06.847785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.847764 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365d84dc-cf64-4d85-9d5b-05647ab37fd1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7\" (UID: \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:06.851125 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.850694 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365d84dc-cf64-4d85-9d5b-05647ab37fd1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7\" (UID: \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:06.907747 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.907636 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcpjp\" (UniqueName: \"kubernetes.io/projected/365d84dc-cf64-4d85-9d5b-05647ab37fd1-kube-api-access-xcpjp\") pod \"payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7\" (UID: \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:06.952116 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:06.952085 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:54:07.052115 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.052083 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2989c9-8271-45bd-91a5-e54e008e14de-kube-api-access-8f9lh\" (UniqueName: \"kubernetes.io/projected/af2989c9-8271-45bd-91a5-e54e008e14de-kube-api-access-8f9lh\") pod \"af2989c9-8271-45bd-91a5-e54e008e14de\" (UID: \"af2989c9-8271-45bd-91a5-e54e008e14de\") " Nov 02 19:54:07.052321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.052147 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2989c9-8271-45bd-91a5-e54e008e14de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2989c9-8271-45bd-91a5-e54e008e14de-plugins-config-volume\") pod \"af2989c9-8271-45bd-91a5-e54e008e14de\" (UID: \"af2989c9-8271-45bd-91a5-e54e008e14de\") " Nov 02 19:54:07.052321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.052285 5850 generic.go:356] "Generic (PLEG): container finished" podID="af2989c9-8271-45bd-91a5-e54e008e14de" containerID="ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad" exitCode=0 Nov 02 19:54:07.052435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.052330 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" Nov 02 19:54:07.052435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.052368 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" event={"ID":"af2989c9-8271-45bd-91a5-e54e008e14de","Type":"ContainerDied","Data":"ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad"} Nov 02 19:54:07.052435 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.052425 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9" event={"ID":"af2989c9-8271-45bd-91a5-e54e008e14de","Type":"ContainerDied","Data":"2f70b160a9e9083d6db31f180eedde67989a8ce40096e9f2c40e0345f25ee3e8"} Nov 02 19:54:07.052648 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.052455 5850 scope.go:122] "RemoveContainer" containerID="ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad" Nov 02 19:54:07.052648 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.052576 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2989c9-8271-45bd-91a5-e54e008e14de-plugins-config-volume" pod "af2989c9-8271-45bd-91a5-e54e008e14de" (UID: "af2989c9-8271-45bd-91a5-e54e008e14de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:07.052961 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.052944 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2989c9-8271-45bd-91a5-e54e008e14de-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:07.054441 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.054418 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2989c9-8271-45bd-91a5-e54e008e14de-kube-api-access-8f9lh" pod "af2989c9-8271-45bd-91a5-e54e008e14de" (UID: "af2989c9-8271-45bd-91a5-e54e008e14de"). InnerVolumeSpecName "kube-api-access-8f9lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:07.063134 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.063113 5850 scope.go:122] "RemoveContainer" containerID="ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad" Nov 02 19:54:07.063431 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:07.063412 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad\": container with ID starting with ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad not found: ID does not exist" containerID="ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad" Nov 02 19:54:07.063491 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.063439 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad"} err="failed to get container status \"ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad\": rpc error: code = NotFound desc = could not find container \"ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad\": container with ID starting with ba960d7b5e8cbe74c4f763a5138dac41adb172d9fe91479a0d137c640ff729ad not found: ID does not exist" Nov 02 19:54:07.137075 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.137043 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:07.154816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.154785 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f9lh\" (UniqueName: \"kubernetes.io/projected/af2989c9-8271-45bd-91a5-e54e008e14de-kube-api-access-8f9lh\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:07.205400 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.202607 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq"] Nov 02 19:54:07.207897 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:54:07.207859 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba308b4_8fec_4518_a5c3_245ed4f543f7.slice/crio-29fe50273f1469a40689854cc8d75f48de485d05401b93dce2f22fc9fe39ce78 WatchSource:0}: Error finding container 29fe50273f1469a40689854cc8d75f48de485d05401b93dce2f22fc9fe39ce78: Status 404 returned error can't find the container with id 29fe50273f1469a40689854cc8d75f48de485d05401b93dce2f22fc9fe39ce78 Nov 02 19:54:07.477826 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.477721 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9"] Nov 02 19:54:07.481488 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.481448 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46f86d84-7d6df68d7f-hmns9"] Nov 02 19:54:07.525174 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:54:07.525113 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365d84dc_cf64_4d85_9d5b_05647ab37fd1.slice/crio-f7074b36ff7c8d1947557c7a8ec15ee935d66ac8fef2424934d1b92635208631 WatchSource:0}: Error finding container f7074b36ff7c8d1947557c7a8ec15ee935d66ac8fef2424934d1b92635208631: Status 404 returned error can't find the container with id f7074b36ff7c8d1947557c7a8ec15ee935d66ac8fef2424934d1b92635208631 Nov 02 19:54:07.528690 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.528668 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7"] Nov 02 19:54:07.816893 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:07.816817 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2989c9-8271-45bd-91a5-e54e008e14de" path="/var/lib/kubelet/pods/af2989c9-8271-45bd-91a5-e54e008e14de/volumes" Nov 02 19:54:08.062966 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.062921 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" event={"ID":"9ba308b4-8fec-4518-a5c3-245ed4f543f7","Type":"ContainerStarted","Data":"03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a"} Nov 02 19:54:08.063163 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.062988 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:08.063163 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.063013 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" event={"ID":"9ba308b4-8fec-4518-a5c3-245ed4f543f7","Type":"ContainerStarted","Data":"29fe50273f1469a40689854cc8d75f48de485d05401b93dce2f22fc9fe39ce78"} Nov 02 19:54:08.064670 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.064643 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" event={"ID":"365d84dc-cf64-4d85-9d5b-05647ab37fd1","Type":"ContainerStarted","Data":"fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9"} Nov 02 19:54:08.064781 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.064704 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:08.064781 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.064731 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" event={"ID":"365d84dc-cf64-4d85-9d5b-05647ab37fd1","Type":"ContainerStarted","Data":"f7074b36ff7c8d1947557c7a8ec15ee935d66ac8fef2424934d1b92635208631"} Nov 02 19:54:08.181292 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.181235 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" podStartSLOduration=2.181216908 podStartE2EDuration="2.181216908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:08.180036414 +0000 UTC m=+1506.522425494" watchObservedRunningTime="2026-11-02 19:54:08.181216908 +0000 UTC m=+1506.523605989" Nov 02 19:54:08.187786 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.187754 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg"] Nov 02 19:54:08.188483 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.188454 5850 cpu_manager.go:414] "containerMap: removing container" podUID="af2989c9-8271-45bd-91a5-e54e008e14de" containerName="payload-processing" Nov 02 19:54:08.188538 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.188484 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2989c9-8271-45bd-91a5-e54e008e14de" containerName="payload-processing" Nov 02 19:54:08.188664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.188654 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2989c9-8271-45bd-91a5-e54e008e14de" containerName="payload-processing" Nov 02 19:54:08.191814 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.191794 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.245211 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.245181 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4099cbd3-serving-cert\"" Nov 02 19:54:08.271565 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.271523 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x6zg\" (UniqueName: \"kubernetes.io/projected/da5f54e7-06b6-4e0f-8d71-b24aae87c630-kube-api-access-7x6zg\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.271749 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.271601 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-metrics-certs\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.271749 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.271660 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5f54e7-06b6-4e0f-8d71-b24aae87c630-openshift-service-ca\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.271749 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.271732 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.311886 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.290548 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg"] Nov 02 19:54:08.333496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.324773 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" podStartSLOduration=2.324752401 podStartE2EDuration="2.324752401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:08.322688493 +0000 UTC m=+1506.665077571" watchObservedRunningTime="2026-11-02 19:54:08.324752401 +0000 UTC m=+1506.667141482" Nov 02 19:54:08.386496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.383296 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x6zg\" (UniqueName: \"kubernetes.io/projected/da5f54e7-06b6-4e0f-8d71-b24aae87c630-kube-api-access-7x6zg\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.386496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.383372 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-metrics-certs\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.386496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.383403 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5f54e7-06b6-4e0f-8d71-b24aae87c630-openshift-service-ca\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.386496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.383452 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.401111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.401059 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:08.402746 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:08.402652 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-serving-cert: secret "maas-api-e2e-mt-4099cbd3-serving-cert" not found Nov 02 19:54:08.402746 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:08.402744 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls podName:da5f54e7-06b6-4e0f-8d71-b24aae87c630 nodeName:}" failed. No retries permitted until 2026-11-02 19:54:08.902718601 +0000 UTC m=+1507.245107677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls") pod "maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" (UID: "da5f54e7-06b6-4e0f-8d71-b24aae87c630") : secret "maas-api-e2e-mt-4099cbd3-serving-cert" not found Nov 02 19:54:08.412060 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.403714 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5f54e7-06b6-4e0f-8d71-b24aae87c630-openshift-service-ca\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.464126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.452790 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-metrics-certs\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.464126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.459033 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x6zg\" (UniqueName: \"kubernetes.io/projected/da5f54e7-06b6-4e0f-8d71-b24aae87c630-kube-api-access-7x6zg\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.559367 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.559333 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:08.758345 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.758262 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:08.915750 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.915717 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:08.918368 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:08.918338 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls\") pod \"maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:09.105929 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:09.105839 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:09.483271 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:54:09.483236 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5f54e7_06b6_4e0f_8d71_b24aae87c630.slice/crio-9fd19ac3b30d36e4d817b496f983f707edbbf1fd70a74c39c8d177bbe79c4bec WatchSource:0}: Error finding container 9fd19ac3b30d36e4d817b496f983f707edbbf1fd70a74c39c8d177bbe79c4bec: Status 404 returned error can't find the container with id 9fd19ac3b30d36e4d817b496f983f707edbbf1fd70a74c39c8d177bbe79c4bec Nov 02 19:54:09.486202 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:09.486179 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg"] Nov 02 19:54:10.079911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:10.079876 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" event={"ID":"da5f54e7-06b6-4e0f-8d71-b24aae87c630","Type":"ContainerStarted","Data":"9fd19ac3b30d36e4d817b496f983f707edbbf1fd70a74c39c8d177bbe79c4bec"} Nov 02 19:54:11.102528 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:11.102411 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" event={"ID":"da5f54e7-06b6-4e0f-8d71-b24aae87c630","Type":"ContainerStarted","Data":"0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d"} Nov 02 19:54:11.104540 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:11.104453 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:11.221982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:11.221928 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" podStartSLOduration=2.343103168 podStartE2EDuration="3.221912511s" totalImagesPullingTime="878.809343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:11.221297677 +0000 UTC m=+1509.563686757" watchObservedRunningTime="2026-11-02 19:54:11.221912511 +0000 UTC m=+1509.564301590" Nov 02 19:54:11.931892 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:11.931854 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:12.472968 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:12.472928 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:12.921916 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:12.921883 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:14.764766 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:14.764726 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:54:17.067048 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.067011 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg"] Nov 02 19:54:17.070944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.070924 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.130454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.130423 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg"] Nov 02 19:54:17.166228 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.166187 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.166434 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.166347 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.166575 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.166530 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b5dd6-c428-432f-a1db-b60e6aecb665-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.166735 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.166715 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58blk\" (UniqueName: \"kubernetes.io/projected/1d9b5dd6-c428-432f-a1db-b60e6aecb665-kube-api-access-58blk\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.176614 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.176580 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-4fd0d5-serving-cert\"" Nov 02 19:54:17.268228 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.268198 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.268411 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.268308 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b5dd6-c428-432f-a1db-b60e6aecb665-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.268454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.268432 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58blk\" (UniqueName: \"kubernetes.io/projected/1d9b5dd6-c428-432f-a1db-b60e6aecb665-kube-api-access-58blk\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.268638 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.268620 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.268778 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:17.268758 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-serving-cert: secret "maas-api-e2e-shared-a-w5-4fd0d5-serving-cert" not found Nov 02 19:54:17.268849 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:17.268839 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls podName:1d9b5dd6-c428-432f-a1db-b60e6aecb665 nodeName:}" failed. No retries permitted until 2026-11-02 19:54:17.768814713 +0000 UTC m=+1516.111203774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls") pod "maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" (UID: "1d9b5dd6-c428-432f-a1db-b60e6aecb665") : secret "maas-api-e2e-shared-a-w5-4fd0d5-serving-cert" not found Nov 02 19:54:17.269080 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.269063 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b5dd6-c428-432f-a1db-b60e6aecb665-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.270724 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.270705 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.271903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.271882 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:17.353546 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.347360 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58blk\" (UniqueName: \"kubernetes.io/projected/1d9b5dd6-c428-432f-a1db-b60e6aecb665-kube-api-access-58blk\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.450497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.443880 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:17.607006 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.606916 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:17.786822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.786786 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.789414 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.789310 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:17.984965 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:17.984928 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:18.124147 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:18.124113 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:54:18.370836 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:18.370804 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg"] Nov 02 19:54:18.373409 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:54:18.373260 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9b5dd6_c428_432f_a1db_b60e6aecb665.slice/crio-a7f3d8b9cdd34e4455632858d8178d84ee22e7012243b827d26d9da9c62e3fd4 WatchSource:0}: Error finding container a7f3d8b9cdd34e4455632858d8178d84ee22e7012243b827d26d9da9c62e3fd4: Status 404 returned error can't find the container with id a7f3d8b9cdd34e4455632858d8178d84ee22e7012243b827d26d9da9c62e3fd4 Nov 02 19:54:19.074923 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:19.074895 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 19:54:19.075106 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:19.074970 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 19:54:19.169135 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:19.169096 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" event={"ID":"1d9b5dd6-c428-432f-a1db-b60e6aecb665","Type":"ContainerStarted","Data":"a7f3d8b9cdd34e4455632858d8178d84ee22e7012243b827d26d9da9c62e3fd4"} Nov 02 19:54:20.178010 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:20.177960 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" event={"ID":"1d9b5dd6-c428-432f-a1db-b60e6aecb665","Type":"ContainerStarted","Data":"01f1e99d1f88cacb579ef7dbd837f74328c972774e28ded3e6422fa42c9d0185"} Nov 02 19:54:20.178447 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:20.178054 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:20.296617 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:20.296552 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" podStartSLOduration=2.45809631 podStartE2EDuration="3.29653655s" totalImagesPullingTime="838.44024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:20.293892083 +0000 UTC m=+1518.636281176" watchObservedRunningTime="2026-11-02 19:54:20.29653655 +0000 UTC m=+1518.638925629" Nov 02 19:54:21.797845 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:21.797810 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:26.173902 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:26.173867 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:26.219045 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:26.219009 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 19:54:28.639448 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:28.639394 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:29.857300 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:29.857260 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57"] Nov 02 19:54:29.857741 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:29.857491 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" podUID="55986bdf-983a-4ecb-a0a4-fd785a06066d" containerName="maas-api" containerID="cri-o://be6ffcf4eeea40c9a386a334380bc7d3239e3c3424d06d784ce12a448bced1a7" gracePeriod=30 Nov 02 19:54:29.971520 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:29.971428 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:30.372971 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.372432 5850 generic.go:356] "Generic (PLEG): container finished" podID="55986bdf-983a-4ecb-a0a4-fd785a06066d" containerID="be6ffcf4eeea40c9a386a334380bc7d3239e3c3424d06d784ce12a448bced1a7" exitCode=0 Nov 02 19:54:30.372971 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.372596 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" event={"ID":"55986bdf-983a-4ecb-a0a4-fd785a06066d","Type":"ContainerDied","Data":"be6ffcf4eeea40c9a386a334380bc7d3239e3c3424d06d784ce12a448bced1a7"} Nov 02 19:54:30.378570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.378190 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:54:30.502221 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.502186 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:30.506459 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.506427 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-metrics-certs\") pod \"55986bdf-983a-4ecb-a0a4-fd785a06066d\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " Nov 02 19:54:30.506785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.506761 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55986bdf-983a-4ecb-a0a4-fd785a06066d-kube-api-access-kv2c5\" (UniqueName: \"kubernetes.io/projected/55986bdf-983a-4ecb-a0a4-fd785a06066d-kube-api-access-kv2c5\") pod \"55986bdf-983a-4ecb-a0a4-fd785a06066d\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " Nov 02 19:54:30.507043 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.507025 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-maas-api-tls\") pod \"55986bdf-983a-4ecb-a0a4-fd785a06066d\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " Nov 02 19:54:30.507432 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.507408 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55986bdf-983a-4ecb-a0a4-fd785a06066d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55986bdf-983a-4ecb-a0a4-fd785a06066d-openshift-service-ca\") pod \"55986bdf-983a-4ecb-a0a4-fd785a06066d\" (UID: \"55986bdf-983a-4ecb-a0a4-fd785a06066d\") " Nov 02 19:54:30.511406 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.511378 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55986bdf-983a-4ecb-a0a4-fd785a06066d-openshift-service-ca" pod "55986bdf-983a-4ecb-a0a4-fd785a06066d" (UID: "55986bdf-983a-4ecb-a0a4-fd785a06066d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:30.529654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.528742 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-metrics-certs" pod "55986bdf-983a-4ecb-a0a4-fd785a06066d" (UID: "55986bdf-983a-4ecb-a0a4-fd785a06066d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:54:30.533113 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.530034 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-maas-api-tls" pod "55986bdf-983a-4ecb-a0a4-fd785a06066d" (UID: "55986bdf-983a-4ecb-a0a4-fd785a06066d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:54:30.534424 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.534388 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55986bdf-983a-4ecb-a0a4-fd785a06066d-kube-api-access-kv2c5" pod "55986bdf-983a-4ecb-a0a4-fd785a06066d" (UID: "55986bdf-983a-4ecb-a0a4-fd785a06066d"). InnerVolumeSpecName "kube-api-access-kv2c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:30.611131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.609920 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55986bdf-983a-4ecb-a0a4-fd785a06066d-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:30.611131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.609958 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:30.611131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.609971 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv2c5\" (UniqueName: \"kubernetes.io/projected/55986bdf-983a-4ecb-a0a4-fd785a06066d-kube-api-access-kv2c5\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:30.611131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:30.609980 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55986bdf-983a-4ecb-a0a4-fd785a06066d-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:31.381626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:31.381593 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" Nov 02 19:54:31.382112 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:31.381598 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57" event={"ID":"55986bdf-983a-4ecb-a0a4-fd785a06066d","Type":"ContainerDied","Data":"6513c23c56cad6c03ff0919f1347df208b0e775f64379d6c23b40fcd1eacc301"} Nov 02 19:54:31.382112 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:31.381724 5850 scope.go:122] "RemoveContainer" containerID="be6ffcf4eeea40c9a386a334380bc7d3239e3c3424d06d784ce12a448bced1a7" Nov 02 19:54:31.392822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:31.392797 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:31.552040 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:31.552008 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57"] Nov 02 19:54:31.604094 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:31.604051 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d368a053-6dddbf54dc-rbs57"] Nov 02 19:54:31.632005 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:31.631918 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:31.782209 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:31.780681 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:31.842542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:31.842511 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55986bdf-983a-4ecb-a0a4-fd785a06066d" path="/var/lib/kubelet/pods/55986bdf-983a-4ecb-a0a4-fd785a06066d/volumes" Nov 02 19:54:33.425734 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:33.425690 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:34.250115 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:34.250081 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:34.489218 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:34.489186 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:34.742376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:34.742329 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv"] Nov 02 19:54:34.743150 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:34.743132 5850 cpu_manager.go:414] "containerMap: removing container" podUID="55986bdf-983a-4ecb-a0a4-fd785a06066d" containerName="maas-api" Nov 02 19:54:34.743212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:34.743151 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55986bdf-983a-4ecb-a0a4-fd785a06066d" containerName="maas-api" Nov 02 19:54:34.743262 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:34.743253 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="55986bdf-983a-4ecb-a0a4-fd785a06066d" containerName="maas-api" Nov 02 19:54:34.747098 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:34.747078 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:34.917598 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:34.906837 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-d1aaed-openshift-default-dockercfg-qmhvl\"" Nov 02 19:54:34.917598 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:34.912485 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:35.050686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.049662 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-podinfo\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.050686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.049713 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-socket\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.050686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.049755 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-data\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.050686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.049942 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-credential-socket\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.050686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.049992 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-certs\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.050686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.050107 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-envoy\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.050686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.050186 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb4481-97cd-44e3-a14a-01dbcac2752c-istiod-ca-cert\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.050686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.050215 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-token\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.050686 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.050241 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vfwh\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-kube-api-access-8vfwh\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.063513 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.061557 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv"] Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.151768 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-credential-socket\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.151832 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-certs\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.151938 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-envoy\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.152004 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb4481-97cd-44e3-a14a-01dbcac2752c-istiod-ca-cert\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.152031 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-token\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.152057 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vfwh\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-kube-api-access-8vfwh\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.152162 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-podinfo\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.152226 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-socket\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.152266 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-data\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.152759 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-data\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.152978 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-credential-socket\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.153535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.153208 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-certs\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.161232 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.154591 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb4481-97cd-44e3-a14a-01dbcac2752c-istiod-ca-cert\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.161232 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.156866 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-socket\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.161232 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.159875 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-podinfo\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.191497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.185493 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-envoy\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.229706 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.229670 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:35.250547 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.247261 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vfwh\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-kube-api-access-8vfwh\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.291709 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.287139 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-token\") pod \"e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.363916 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.362815 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:35.867521 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:54:35.867490 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cb4481_97cd_44e3_a14a_01dbcac2752c.slice/crio-af93fd83df8b8e2b8c88d712b0fe99e20c3bbb629450565f8248d7c011264189 WatchSource:0}: Error finding container af93fd83df8b8e2b8c88d712b0fe99e20c3bbb629450565f8248d7c011264189: Status 404 returned error can't find the container with id af93fd83df8b8e2b8c88d712b0fe99e20c3bbb629450565f8248d7c011264189 Nov 02 19:54:35.870681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.870646 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:54:35.870773 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.870731 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:54:35.870773 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.870761 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:54:35.871522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.871152 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv"] Nov 02 19:54:35.966132 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:35.966107 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:36.554042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:36.553989 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" event={"ID":"71cb4481-97cd-44e3-a14a-01dbcac2752c","Type":"ContainerStarted","Data":"7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54"} Nov 02 19:54:36.554352 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:36.554327 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" event={"ID":"71cb4481-97cd-44e3-a14a-01dbcac2752c","Type":"ContainerStarted","Data":"af93fd83df8b8e2b8c88d712b0fe99e20c3bbb629450565f8248d7c011264189"} Nov 02 19:54:36.676621 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:36.676561 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" podStartSLOduration=2.676543753 podStartE2EDuration="2.676543753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:36.673851537 +0000 UTC m=+1535.016240618" watchObservedRunningTime="2026-11-02 19:54:36.676543753 +0000 UTC m=+1535.018932833" Nov 02 19:54:37.117497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:37.116775 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:37.363673 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:37.363634 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:37.369908 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:37.369830 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:37.532276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:37.532235 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:37.561313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:37.561275 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:37.562323 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:37.562298 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:54:37.797510 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:37.797399 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:37.973681 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:37.973635 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:38.048181 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:38.048097 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz"] Nov 02 19:54:38.048373 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:38.048346 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" containerID="cri-o://a6f9c228f414c01b519129da8e20b378998a76043c719fb23ff616a1b77efa2a" gracePeriod=30 Nov 02 19:54:38.135269 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:38.135234 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:39.644687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:39.644608 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:39.813399 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:39.813358 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:39.975025 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:39.974930 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:42.888335 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:42.888301 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:43.050974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:43.050768 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:43.279611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:43.279509 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:43.802241 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:43.801786 5850 generic.go:356] "Generic (PLEG): container finished" podID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerID="a6f9c228f414c01b519129da8e20b378998a76043c719fb23ff616a1b77efa2a" exitCode=0 Nov 02 19:54:43.802241 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:43.801951 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" event={"ID":"8ab6ece9-b656-4a56-ad6e-1a3299cb239b","Type":"ContainerDied","Data":"a6f9c228f414c01b519129da8e20b378998a76043c719fb23ff616a1b77efa2a"} Nov 02 19:54:43.802241 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:43.802068 5850 scope.go:122] "RemoveContainer" containerID="547194ff78096e8eaf61179ad25e071976443fdff5bee0c2001d7758813fc4c7" Nov 02 19:54:43.989923 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:43.989889 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:54:44.138649 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.138595 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-socket\") pod \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " Nov 02 19:54:44.138848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.138797 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-certs\") pod \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " Nov 02 19:54:44.138928 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.138883 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-socket" pod "8ab6ece9-b656-4a56-ad6e-1a3299cb239b" (UID: "8ab6ece9-b656-4a56-ad6e-1a3299cb239b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:44.138998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.138951 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-envoy\") pod \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " Nov 02 19:54:44.138998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.138991 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istiod-ca-cert\") pod \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " Nov 02 19:54:44.139102 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.139060 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-data\") pod \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " Nov 02 19:54:44.139153 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.139110 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-certs" pod "8ab6ece9-b656-4a56-ad6e-1a3299cb239b" (UID: "8ab6ece9-b656-4a56-ad6e-1a3299cb239b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:44.139153 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.139125 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-credential-socket\") pod \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " Nov 02 19:54:44.139311 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.139157 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-podinfo\") pod \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " Nov 02 19:54:44.139311 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.139221 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-token\") pod \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " Nov 02 19:54:44.139311 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.139255 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-kube-api-access-wbcnx\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-kube-api-access-wbcnx\") pod \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\" (UID: \"8ab6ece9-b656-4a56-ad6e-1a3299cb239b\") " Nov 02 19:54:44.139457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.139312 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-data" pod "8ab6ece9-b656-4a56-ad6e-1a3299cb239b" (UID: "8ab6ece9-b656-4a56-ad6e-1a3299cb239b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:44.139457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.139401 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istiod-ca-cert" pod "8ab6ece9-b656-4a56-ad6e-1a3299cb239b" (UID: "8ab6ece9-b656-4a56-ad6e-1a3299cb239b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:44.139457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.139447 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-credential-socket" pod "8ab6ece9-b656-4a56-ad6e-1a3299cb239b" (UID: "8ab6ece9-b656-4a56-ad6e-1a3299cb239b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:44.140038 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.140019 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:44.140038 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.140040 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:44.140193 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.140050 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:44.140193 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.140059 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:44.140193 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.140072 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:44.141784 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.141747 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-kube-api-access-wbcnx" pod "8ab6ece9-b656-4a56-ad6e-1a3299cb239b" (UID: "8ab6ece9-b656-4a56-ad6e-1a3299cb239b"). InnerVolumeSpecName "kube-api-access-wbcnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:44.141885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.141834 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-podinfo" pod "8ab6ece9-b656-4a56-ad6e-1a3299cb239b" (UID: "8ab6ece9-b656-4a56-ad6e-1a3299cb239b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:54:44.141941 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.141884 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-envoy" pod "8ab6ece9-b656-4a56-ad6e-1a3299cb239b" (UID: "8ab6ece9-b656-4a56-ad6e-1a3299cb239b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:44.141997 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.141951 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-token" pod "8ab6ece9-b656-4a56-ad6e-1a3299cb239b" (UID: "8ab6ece9-b656-4a56-ad6e-1a3299cb239b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:44.241654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.241609 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:44.241654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.241648 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:44.241654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.241661 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:44.241897 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.241675 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbcnx\" (UniqueName: \"kubernetes.io/projected/8ab6ece9-b656-4a56-ad6e-1a3299cb239b-kube-api-access-wbcnx\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:44.765707 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:44.765666 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:54:44.811444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.811412 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" Nov 02 19:54:44.811444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.811420 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz" event={"ID":"8ab6ece9-b656-4a56-ad6e-1a3299cb239b","Type":"ContainerDied","Data":"89ded070c07fdc98747663b8ed1f6b142b5408a4657967c91a529d38b2fb1a2c"} Nov 02 19:54:44.811692 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:44.811502 5850 scope.go:122] "RemoveContainer" containerID="a6f9c228f414c01b519129da8e20b378998a76043c719fb23ff616a1b77efa2a" Nov 02 19:54:45.101153 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:45.101058 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz"] Nov 02 19:54:45.106209 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:45.106183 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-46f86d84-openshift-default-64cd95d87c-dzgmz"] Nov 02 19:54:45.817658 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:45.817624 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" path="/var/lib/kubelet/pods/8ab6ece9-b656-4a56-ad6e-1a3299cb239b/volumes" Nov 02 19:54:47.542203 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:47.542174 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:51.097307 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.097271 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b"] Nov 02 19:54:51.097866 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.097646 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" containerID="cri-o://84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e" gracePeriod=30 Nov 02 19:54:51.149523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.149466 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8"] Nov 02 19:54:51.149731 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.149690 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" containerID="cri-o://60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513" gracePeriod=30 Nov 02 19:54:51.153779 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.153750 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh"] Nov 02 19:54:51.153956 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.153938 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" containerID="cri-o://a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1" gracePeriod=30 Nov 02 19:54:51.199356 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.199323 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls"] Nov 02 19:54:51.199590 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.199554 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" containerID="cri-o://6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf" gracePeriod=30 Nov 02 19:54:51.599036 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.598997 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7"] Nov 02 19:54:51.599726 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.599712 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" Nov 02 19:54:51.599775 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.599728 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" Nov 02 19:54:51.599882 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.599873 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" Nov 02 19:54:51.599921 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.599884 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" Nov 02 19:54:51.602972 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.602951 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.657203 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.657171 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-282875-serving-cert\"" Nov 02 19:54:51.661348 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.661312 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7"] Nov 02 19:54:51.681550 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.681464 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680240bb-879b-46bf-b773-78cd4e768a26-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.681729 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.681641 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.681729 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.681688 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.681887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.681862 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bs4d\" (UniqueName: \"kubernetes.io/projected/680240bb-879b-46bf-b773-78cd4e768a26-kube-api-access-5bs4d\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.783203 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.783164 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680240bb-879b-46bf-b773-78cd4e768a26-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.783378 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.783241 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.783378 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.783265 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.783378 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.783324 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bs4d\" (UniqueName: \"kubernetes.io/projected/680240bb-879b-46bf-b773-78cd4e768a26-kube-api-access-5bs4d\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.783534 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:51.783404 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-serving-cert: secret "maas-api-e2e-shared-b-w1-282875-serving-cert" not found Nov 02 19:54:51.783534 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:51.783511 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls podName:680240bb-879b-46bf-b773-78cd4e768a26 nodeName:}" failed. No retries permitted until 2026-11-02 19:54:52.283484859 +0000 UTC m=+1550.625873932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls") pod "maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" (UID: "680240bb-879b-46bf-b773-78cd4e768a26") : secret "maas-api-e2e-shared-b-w1-282875-serving-cert" not found Nov 02 19:54:51.783908 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.783886 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680240bb-879b-46bf-b773-78cd4e768a26-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.785849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.785820 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:51.796062 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.796035 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:51.842268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:51.842232 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bs4d\" (UniqueName: \"kubernetes.io/projected/680240bb-879b-46bf-b773-78cd4e768a26-kube-api-access-5bs4d\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:52.026494 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:52.024006 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:52.195813 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:52.195382 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:52.316771 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:52.316680 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:52.319279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:52.319247 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:52.518196 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:52.518154 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:52.890633 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:54:52.890440 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680240bb_879b_46bf_b773_78cd4e768a26.slice/crio-ff607c80c942805b6fe792a4e0b4b2f5ee871b1d614ed4d9e0f01b5d1dacfce2 WatchSource:0}: Error finding container ff607c80c942805b6fe792a4e0b4b2f5ee871b1d614ed4d9e0f01b5d1dacfce2: Status 404 returned error can't find the container with id ff607c80c942805b6fe792a4e0b4b2f5ee871b1d614ed4d9e0f01b5d1dacfce2 Nov 02 19:54:52.895603 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:52.895572 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7"] Nov 02 19:54:53.880937 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:53.880900 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" event={"ID":"680240bb-879b-46bf-b773-78cd4e768a26","Type":"ContainerStarted","Data":"941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79"} Nov 02 19:54:53.881425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:53.880960 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" event={"ID":"680240bb-879b-46bf-b773-78cd4e768a26","Type":"ContainerStarted","Data":"ff607c80c942805b6fe792a4e0b4b2f5ee871b1d614ed4d9e0f01b5d1dacfce2"} Nov 02 19:54:53.881425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:53.880988 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:54:54.039038 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:54.038915 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" podStartSLOduration=2.19597996 podStartE2EDuration="3.038895991s" totalImagesPullingTime="842.916031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:54.038108114 +0000 UTC m=+1552.380497194" watchObservedRunningTime="2026-11-02 19:54:54.038895991 +0000 UTC m=+1552.381285070" Nov 02 19:54:56.443358 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.443318 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:56.588854 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.588770 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:54:56.603634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.603596 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:54:56.697285 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.697258 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:54:56.706375 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.706350 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:54:56.724355 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724312 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-token\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-token\") pod \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " Nov 02 19:54:56.724573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724369 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-certs\") pod \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " Nov 02 19:54:56.724573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724389 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-certs\") pod \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " Nov 02 19:54:56.724573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724464 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-kube-api-access-2552h\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-kube-api-access-2552h\") pod \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " Nov 02 19:54:56.724573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724527 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-podinfo\") pod \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " Nov 02 19:54:56.724778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724616 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-socket\") pod \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " Nov 02 19:54:56.724778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724657 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istiod-ca-cert\") pod \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " Nov 02 19:54:56.724778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724726 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-credential-socket\") pod \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " Nov 02 19:54:56.724778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724724 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-certs" pod "0ced20f2-def6-42a7-9c08-96345ef8a5a1" (UID: "0ced20f2-def6-42a7-9c08-96345ef8a5a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.724778 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724739 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-certs" pod "02f750f1-28c2-4d1c-a92e-00bb449e1d3b" (UID: "02f750f1-28c2-4d1c-a92e-00bb449e1d3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.724998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724825 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-token\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-token\") pod \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " Nov 02 19:54:56.724998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724860 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-podinfo\") pod \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " Nov 02 19:54:56.724998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724896 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-credential-socket\") pod \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " Nov 02 19:54:56.724998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724932 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istiod-ca-cert\") pod \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " Nov 02 19:54:56.724998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724950 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-socket" pod "0ced20f2-def6-42a7-9c08-96345ef8a5a1" (UID: "0ced20f2-def6-42a7-9c08-96345ef8a5a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.724998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.724958 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-envoy\") pod \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " Nov 02 19:54:56.725267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.725004 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-data\") pod \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " Nov 02 19:54:56.725267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.725067 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-credential-socket" pod "02f750f1-28c2-4d1c-a92e-00bb449e1d3b" (UID: "02f750f1-28c2-4d1c-a92e-00bb449e1d3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.725267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.725095 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istiod-ca-cert" pod "02f750f1-28c2-4d1c-a92e-00bb449e1d3b" (UID: "02f750f1-28c2-4d1c-a92e-00bb449e1d3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:56.725267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.725146 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-envoy\") pod \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " Nov 02 19:54:56.725267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.725171 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-credential-socket" pod "0ced20f2-def6-42a7-9c08-96345ef8a5a1" (UID: "0ced20f2-def6-42a7-9c08-96345ef8a5a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.725267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.725207 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-data\") pod \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " Nov 02 19:54:56.725578 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.725281 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-socket\") pod \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\" (UID: \"02f750f1-28c2-4d1c-a92e-00bb449e1d3b\") " Nov 02 19:54:56.725578 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.725340 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-kube-api-access-sbjlq\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-kube-api-access-sbjlq\") pod \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\" (UID: \"0ced20f2-def6-42a7-9c08-96345ef8a5a1\") " Nov 02 19:54:56.726328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.726302 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.726452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.726363 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.726452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.726380 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.726452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.726395 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.726452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.726408 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.726452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.726439 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.727608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.727063 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-data" pod "02f750f1-28c2-4d1c-a92e-00bb449e1d3b" (UID: "02f750f1-28c2-4d1c-a92e-00bb449e1d3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.727608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.727071 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-data" pod "0ced20f2-def6-42a7-9c08-96345ef8a5a1" (UID: "0ced20f2-def6-42a7-9c08-96345ef8a5a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.727608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.727077 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-socket" pod "02f750f1-28c2-4d1c-a92e-00bb449e1d3b" (UID: "02f750f1-28c2-4d1c-a92e-00bb449e1d3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.727608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.727161 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istiod-ca-cert" pod "0ced20f2-def6-42a7-9c08-96345ef8a5a1" (UID: "0ced20f2-def6-42a7-9c08-96345ef8a5a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:56.727872 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.727668 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-kube-api-access-2552h" pod "02f750f1-28c2-4d1c-a92e-00bb449e1d3b" (UID: "02f750f1-28c2-4d1c-a92e-00bb449e1d3b"). InnerVolumeSpecName "kube-api-access-2552h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:56.728182 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.728156 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-envoy" pod "02f750f1-28c2-4d1c-a92e-00bb449e1d3b" (UID: "02f750f1-28c2-4d1c-a92e-00bb449e1d3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.728281 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.728213 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-envoy" pod "0ced20f2-def6-42a7-9c08-96345ef8a5a1" (UID: "0ced20f2-def6-42a7-9c08-96345ef8a5a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.728405 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.728384 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-token" pod "0ced20f2-def6-42a7-9c08-96345ef8a5a1" (UID: "0ced20f2-def6-42a7-9c08-96345ef8a5a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:56.728458 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.728431 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-token" pod "02f750f1-28c2-4d1c-a92e-00bb449e1d3b" (UID: "02f750f1-28c2-4d1c-a92e-00bb449e1d3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:56.728536 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.728460 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-podinfo" pod "0ced20f2-def6-42a7-9c08-96345ef8a5a1" (UID: "0ced20f2-def6-42a7-9c08-96345ef8a5a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:54:56.728871 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.728855 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-kube-api-access-sbjlq" pod "0ced20f2-def6-42a7-9c08-96345ef8a5a1" (UID: "0ced20f2-def6-42a7-9c08-96345ef8a5a1"). InnerVolumeSpecName "kube-api-access-sbjlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:56.730008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.729989 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-podinfo" pod "02f750f1-28c2-4d1c-a92e-00bb449e1d3b" (UID: "02f750f1-28c2-4d1c-a92e-00bb449e1d3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:54:56.827814 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.827776 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-token\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-token\") pod \"9a0c0942-7045-466b-88b2-a1a72f3afda0\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " Nov 02 19:54:56.827999 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.827868 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0c0942-7045-466b-88b2-a1a72f3afda0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0c0942-7045-466b-88b2-a1a72f3afda0-istiod-ca-cert\") pod \"9a0c0942-7045-466b-88b2-a1a72f3afda0\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " Nov 02 19:54:56.827999 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.827917 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-podinfo\") pod \"9a0c0942-7045-466b-88b2-a1a72f3afda0\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " Nov 02 19:54:56.827999 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.827941 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-envoy\") pod \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " Nov 02 19:54:56.827999 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.827992 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-socket\") pod \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " Nov 02 19:54:56.828212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828069 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-podinfo\") pod \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " Nov 02 19:54:56.828212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828119 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-data\") pod \"9a0c0942-7045-466b-88b2-a1a72f3afda0\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " Nov 02 19:54:56.828212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828145 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-token\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-token\") pod \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " Nov 02 19:54:56.828212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828178 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-data\") pod \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " Nov 02 19:54:56.828438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828247 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-certs\") pod \"9a0c0942-7045-466b-88b2-a1a72f3afda0\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " Nov 02 19:54:56.828438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828332 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-credential-socket\") pod \"9a0c0942-7045-466b-88b2-a1a72f3afda0\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " Nov 02 19:54:56.828438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828371 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-envoy\") pod \"9a0c0942-7045-466b-88b2-a1a72f3afda0\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " Nov 02 19:54:56.828438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828381 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-socket" pod "568ce497-2bab-4ae1-a9cd-3c386dbc28e6" (UID: "568ce497-2bab-4ae1-a9cd-3c386dbc28e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.828438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828395 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0c0942-7045-466b-88b2-a1a72f3afda0-istiod-ca-cert" pod "9a0c0942-7045-466b-88b2-a1a72f3afda0" (UID: "9a0c0942-7045-466b-88b2-a1a72f3afda0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:56.828438 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828404 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-kube-api-access-s6clm\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-kube-api-access-s6clm\") pod \"9a0c0942-7045-466b-88b2-a1a72f3afda0\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " Nov 02 19:54:56.828827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828520 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-kube-api-access-hjmfz\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-kube-api-access-hjmfz\") pod \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " Nov 02 19:54:56.828827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828562 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-credential-socket\") pod \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " Nov 02 19:54:56.828827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828653 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istiod-ca-cert\") pod \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " Nov 02 19:54:56.828827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828663 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-certs" pod "9a0c0942-7045-466b-88b2-a1a72f3afda0" (UID: "9a0c0942-7045-466b-88b2-a1a72f3afda0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.828827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828706 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-socket\") pod \"9a0c0942-7045-466b-88b2-a1a72f3afda0\" (UID: \"9a0c0942-7045-466b-88b2-a1a72f3afda0\") " Nov 02 19:54:56.828827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828744 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-certs\") pod \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\" (UID: \"568ce497-2bab-4ae1-a9cd-3c386dbc28e6\") " Nov 02 19:54:56.829158 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.828854 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-data" pod "568ce497-2bab-4ae1-a9cd-3c386dbc28e6" (UID: "568ce497-2bab-4ae1-a9cd-3c386dbc28e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.829158 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829055 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-data" pod "9a0c0942-7045-466b-88b2-a1a72f3afda0" (UID: "9a0c0942-7045-466b-88b2-a1a72f3afda0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.829369 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829345 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istiod-ca-cert" pod "568ce497-2bab-4ae1-a9cd-3c386dbc28e6" (UID: "568ce497-2bab-4ae1-a9cd-3c386dbc28e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:56.829612 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829586 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-socket" pod "9a0c0942-7045-466b-88b2-a1a72f3afda0" (UID: "9a0c0942-7045-466b-88b2-a1a72f3afda0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.829848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829775 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.829848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829802 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.829848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829818 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.829848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829833 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0c0942-7045-466b-88b2-a1a72f3afda0-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829848 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829864 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbjlq\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-kube-api-access-sbjlq\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829879 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829893 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829908 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2552h\" (UniqueName: \"kubernetes.io/projected/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-kube-api-access-2552h\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829922 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829936 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829950 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829965 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829979 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.829994 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.830008 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f750f1-28c2-4d1c-a92e-00bb449e1d3b-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.830022 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.830039 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ced20f2-def6-42a7-9c08-96345ef8a5a1-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.830056 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.830990 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.830335 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-certs" pod "568ce497-2bab-4ae1-a9cd-3c386dbc28e6" (UID: "568ce497-2bab-4ae1-a9cd-3c386dbc28e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.830990 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.830553 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-credential-socket" pod "568ce497-2bab-4ae1-a9cd-3c386dbc28e6" (UID: "568ce497-2bab-4ae1-a9cd-3c386dbc28e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.830990 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.830691 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-credential-socket" pod "9a0c0942-7045-466b-88b2-a1a72f3afda0" (UID: "9a0c0942-7045-466b-88b2-a1a72f3afda0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.831160 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.831107 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-envoy" pod "568ce497-2bab-4ae1-a9cd-3c386dbc28e6" (UID: "568ce497-2bab-4ae1-a9cd-3c386dbc28e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.831212 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.831177 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-podinfo" pod "9a0c0942-7045-466b-88b2-a1a72f3afda0" (UID: "9a0c0942-7045-466b-88b2-a1a72f3afda0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:54:56.831268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.831212 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-kube-api-access-s6clm" pod "9a0c0942-7045-466b-88b2-a1a72f3afda0" (UID: "9a0c0942-7045-466b-88b2-a1a72f3afda0"). InnerVolumeSpecName "kube-api-access-s6clm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:56.831268 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.831240 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-token" pod "9a0c0942-7045-466b-88b2-a1a72f3afda0" (UID: "9a0c0942-7045-466b-88b2-a1a72f3afda0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:56.831834 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.831812 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-token" pod "568ce497-2bab-4ae1-a9cd-3c386dbc28e6" (UID: "568ce497-2bab-4ae1-a9cd-3c386dbc28e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:56.831988 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.831967 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-kube-api-access-hjmfz" pod "568ce497-2bab-4ae1-a9cd-3c386dbc28e6" (UID: "568ce497-2bab-4ae1-a9cd-3c386dbc28e6"). InnerVolumeSpecName "kube-api-access-hjmfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:56.832315 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.832295 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-podinfo" pod "568ce497-2bab-4ae1-a9cd-3c386dbc28e6" (UID: "568ce497-2bab-4ae1-a9cd-3c386dbc28e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:54:56.832562 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.832541 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-envoy" pod "9a0c0942-7045-466b-88b2-a1a72f3afda0" (UID: "9a0c0942-7045-466b-88b2-a1a72f3afda0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:56.902384 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.902337 5850 generic.go:356] "Generic (PLEG): container finished" podID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerID="a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1" exitCode=0 Nov 02 19:54:56.902644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.902443 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" event={"ID":"568ce497-2bab-4ae1-a9cd-3c386dbc28e6","Type":"ContainerDied","Data":"a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1"} Nov 02 19:54:56.902644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.902510 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" event={"ID":"568ce497-2bab-4ae1-a9cd-3c386dbc28e6","Type":"ContainerDied","Data":"52c733ac778157d9db0720452e72fe5547eba4e5f5e4a9d5300b87b6dcab37a4"} Nov 02 19:54:56.902644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.902543 5850 scope.go:122] "RemoveContainer" containerID="a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1" Nov 02 19:54:56.902822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.902712 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh" Nov 02 19:54:56.906170 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.906142 5850 generic.go:356] "Generic (PLEG): container finished" podID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerID="60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513" exitCode=0 Nov 02 19:54:56.906310 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.906223 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" Nov 02 19:54:56.906310 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.906244 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" event={"ID":"0ced20f2-def6-42a7-9c08-96345ef8a5a1","Type":"ContainerDied","Data":"60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513"} Nov 02 19:54:56.906436 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.906317 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8" event={"ID":"0ced20f2-def6-42a7-9c08-96345ef8a5a1","Type":"ContainerDied","Data":"945f0ea13406f75059d1bf7f2e8aacba028d326dc5100dd74c0030f03c233ade"} Nov 02 19:54:56.908520 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.908492 5850 generic.go:356] "Generic (PLEG): container finished" podID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerID="6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf" exitCode=0 Nov 02 19:54:56.908634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.908556 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" Nov 02 19:54:56.908968 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.908737 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" event={"ID":"9a0c0942-7045-466b-88b2-a1a72f3afda0","Type":"ContainerDied","Data":"6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf"} Nov 02 19:54:56.908968 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.908784 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls" event={"ID":"9a0c0942-7045-466b-88b2-a1a72f3afda0","Type":"ContainerDied","Data":"20f4fe5d5c91367469de77c6c140f64e24c53a4c2ba569f5f8084b4d357fea50"} Nov 02 19:54:56.914745 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.914700 5850 generic.go:356] "Generic (PLEG): container finished" podID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerID="84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e" exitCode=0 Nov 02 19:54:56.914903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.914748 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" event={"ID":"02f750f1-28c2-4d1c-a92e-00bb449e1d3b","Type":"ContainerDied","Data":"84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e"} Nov 02 19:54:56.914903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.914794 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" Nov 02 19:54:56.914903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.914793 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b" event={"ID":"02f750f1-28c2-4d1c-a92e-00bb449e1d3b","Type":"ContainerDied","Data":"bbc16d6f04e2efdc85f1528cf9c09f3b5e6cff2563d5f64eab0714ec2e3d64a4"} Nov 02 19:54:56.922668 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.922493 5850 scope.go:122] "RemoveContainer" containerID="a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1" Nov 02 19:54:56.923386 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:56.923235 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1\": container with ID starting with a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1 not found: ID does not exist" containerID="a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1" Nov 02 19:54:56.923386 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.923273 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1"} err="failed to get container status \"a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1\": rpc error: code = NotFound desc = could not find container \"a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1\": container with ID starting with a498acc08a9f1e3c490edcd05c24cae66abec0a84518457bfbd8e3196658f3f1 not found: ID does not exist" Nov 02 19:54:56.923386 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.923301 5850 scope.go:122] "RemoveContainer" containerID="60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513" Nov 02 19:54:56.931361 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931333 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931364 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931381 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931394 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6clm\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-kube-api-access-s6clm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931408 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjmfz\" (UniqueName: \"kubernetes.io/projected/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-kube-api-access-hjmfz\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931421 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931434 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931448 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931461 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0c0942-7045-466b-88b2-a1a72f3afda0-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931516 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.931816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.931531 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568ce497-2bab-4ae1-a9cd-3c386dbc28e6-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:56.939821 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.939790 5850 scope.go:122] "RemoveContainer" containerID="60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513" Nov 02 19:54:56.940233 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:56.940203 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513\": container with ID starting with 60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513 not found: ID does not exist" containerID="60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513" Nov 02 19:54:56.940328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.940242 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513"} err="failed to get container status \"60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513\": rpc error: code = NotFound desc = could not find container \"60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513\": container with ID starting with 60b34df251a8806df0575d1930a2052c6dc704b0ef0ec5d9942b232f73c56513 not found: ID does not exist" Nov 02 19:54:56.940328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.940264 5850 scope.go:122] "RemoveContainer" containerID="6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf" Nov 02 19:54:56.954437 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.954399 5850 scope.go:122] "RemoveContainer" containerID="e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe" Nov 02 19:54:56.964862 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.964837 5850 scope.go:122] "RemoveContainer" containerID="6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf" Nov 02 19:54:56.965229 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:56.965201 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf\": container with ID starting with 6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf not found: ID does not exist" containerID="6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf" Nov 02 19:54:56.965289 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.965228 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf"} err="failed to get container status \"6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf\": rpc error: code = NotFound desc = could not find container \"6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf\": container with ID starting with 6c0f6a49b2d9752256c1efb30b9d099b0ca7f8573e58f62f708274f5e17a83cf not found: ID does not exist" Nov 02 19:54:56.965289 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.965249 5850 scope.go:122] "RemoveContainer" containerID="e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe" Nov 02 19:54:56.965582 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:56.965553 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe\": container with ID starting with e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe not found: ID does not exist" containerID="e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe" Nov 02 19:54:56.965693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.965592 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe"} err="failed to get container status \"e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe\": rpc error: code = NotFound desc = could not find container \"e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe\": container with ID starting with e39b405c6f7b3862c194e5ab2f9fe7c832fe2b534beb2fbca2466bf91a48f8fe not found: ID does not exist" Nov 02 19:54:56.965693 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.965614 5850 scope.go:122] "RemoveContainer" containerID="84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e" Nov 02 19:54:56.986657 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.986629 5850 scope.go:122] "RemoveContainer" containerID="84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e" Nov 02 19:54:56.987020 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:54:56.986997 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e\": container with ID starting with 84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e not found: ID does not exist" containerID="84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e" Nov 02 19:54:56.987118 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:56.987027 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e"} err="failed to get container status \"84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e\": rpc error: code = NotFound desc = could not find container \"84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e\": container with ID starting with 84d1dde37b069568918f8ee54db88bb9f6d89fa588356f7056e3ee196a14b86e not found: ID does not exist" Nov 02 19:54:57.295569 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.295461 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b"] Nov 02 19:54:57.299067 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.299039 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-qnr9b"] Nov 02 19:54:57.409989 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.409957 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh"] Nov 02 19:54:57.414444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.414409 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-m5dbh"] Nov 02 19:54:57.528077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.528043 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls"] Nov 02 19:54:57.533045 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.533014 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-vblls"] Nov 02 19:54:57.645573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.645540 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8"] Nov 02 19:54:57.649773 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.649745 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-8xwb8"] Nov 02 19:54:57.817372 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.817334 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" path="/var/lib/kubelet/pods/02f750f1-28c2-4d1c-a92e-00bb449e1d3b/volumes" Nov 02 19:54:57.817915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.817900 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" path="/var/lib/kubelet/pods/0ced20f2-def6-42a7-9c08-96345ef8a5a1/volumes" Nov 02 19:54:57.818358 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.818345 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" path="/var/lib/kubelet/pods/568ce497-2bab-4ae1-a9cd-3c386dbc28e6/volumes" Nov 02 19:54:57.819193 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:57.819176 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" path="/var/lib/kubelet/pods/9a0c0942-7045-466b-88b2-a1a72f3afda0/volumes" Nov 02 19:54:59.893604 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:54:59.893576 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 19:55:00.870747 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:00.870717 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:05.222383 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:05.222347 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:09.902520 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:09.902443 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:12.197765 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:12.197713 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:12.798970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:12.798934 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:13.262759 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:13.262725 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:14.223053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:14.222969 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:14.510907 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:14.510822 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:14.664576 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:14.664545 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:14.764396 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:55:14.764309 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:55:14.884856 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:14.884822 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:19.491131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:19.491096 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:21.359706 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:21.359668 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw"] Nov 02 19:55:21.360235 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:21.359901 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" containerID="cri-o://f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a" gracePeriod=30 Nov 02 19:55:21.456703 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:21.456667 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:21.697061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:21.697030 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:21.858135 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:21.858098 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:22.032628 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:22.032538 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:24.306313 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.306228 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j"] Nov 02 19:55:24.307340 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307318 5850 cpu_manager.go:414] "containerMap: removing container" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" Nov 02 19:55:24.307391 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307343 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" Nov 02 19:55:24.307391 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307364 5850 cpu_manager.go:414] "containerMap: removing container" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" Nov 02 19:55:24.307391 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307372 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" Nov 02 19:55:24.307524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307391 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" Nov 02 19:55:24.307524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307398 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab6ece9-b656-4a56-ad6e-1a3299cb239b" containerName="istio-proxy" Nov 02 19:55:24.307524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307447 5850 cpu_manager.go:414] "containerMap: removing container" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" Nov 02 19:55:24.307524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307456 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" Nov 02 19:55:24.307524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307486 5850 cpu_manager.go:414] "containerMap: removing container" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" Nov 02 19:55:24.307524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307493 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" Nov 02 19:55:24.307524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307516 5850 cpu_manager.go:414] "containerMap: removing container" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" Nov 02 19:55:24.307524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307523 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" Nov 02 19:55:24.307772 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307691 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ced20f2-def6-42a7-9c08-96345ef8a5a1" containerName="istio-proxy" Nov 02 19:55:24.307772 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307705 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f750f1-28c2-4d1c-a92e-00bb449e1d3b" containerName="istio-proxy" Nov 02 19:55:24.307772 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307717 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="568ce497-2bab-4ae1-a9cd-3c386dbc28e6" containerName="istio-proxy" Nov 02 19:55:24.307772 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307730 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" Nov 02 19:55:24.307772 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.307740 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0c0942-7045-466b-88b2-a1a72f3afda0" containerName="istio-proxy" Nov 02 19:55:24.312398 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.312368 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.366366 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.366328 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-32abda-openshift-default-dockercfg-rhr6j\"" Nov 02 19:55:24.387117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.387075 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-podinfo\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.387117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.387121 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-envoy\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.387326 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.387224 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-socket\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.387451 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.387434 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-certs\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.387511 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.387484 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istiod-ca-cert\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.387568 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.387517 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-credential-socket\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.387634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.387608 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdbf\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-kube-api-access-gzdbf\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.387788 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.387773 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-token\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.387860 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.387844 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-data\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.410609 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.410572 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j"] Nov 02 19:55:24.416414 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.416377 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.494128 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-data\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.494232 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-podinfo\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.494266 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-envoy\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.494298 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-socket\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.494465 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-certs\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.494514 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istiod-ca-cert\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.494545 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-credential-socket\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.494586 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdbf\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-kube-api-access-gzdbf\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.494827 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-token\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.496502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.495682 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-data\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.507559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.507520 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-podinfo\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.523544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.523506 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-socket\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.528981 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.528944 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-certs\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.534257 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.534217 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istiod-ca-cert\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.534800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.534774 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-credential-socket\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.539496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.535412 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-envoy\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.595682 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.595335 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:24.608568 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.602086 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-token\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.628912 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.628060 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdbf\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-kube-api-access-gzdbf\") pod \"e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.628912 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.628372 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:24.757890 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:24.757809 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:25.141030 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:55:25.140991 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec5c949_3bd9_4ff6_a7be_76c6bf064239.slice/crio-6545dc1ba087d2319de4662df6469117d133bc356499912410f73a6ec13934b5 WatchSource:0}: Error finding container 6545dc1ba087d2319de4662df6469117d133bc356499912410f73a6ec13934b5: Status 404 returned error can't find the container with id 6545dc1ba087d2319de4662df6469117d133bc356499912410f73a6ec13934b5 Nov 02 19:55:25.145126 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:25.145071 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:55:25.145257 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:25.145222 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:55:25.145331 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:25.145306 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:55:25.145915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:25.145890 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j"] Nov 02 19:55:25.327498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:25.325934 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" event={"ID":"cec5c949-3bd9-4ff6-a7be-76c6bf064239","Type":"ContainerStarted","Data":"6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630"} Nov 02 19:55:25.327498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:25.326003 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" event={"ID":"cec5c949-3bd9-4ff6-a7be-76c6bf064239","Type":"ContainerStarted","Data":"6545dc1ba087d2319de4662df6469117d133bc356499912410f73a6ec13934b5"} Nov 02 19:55:25.450521 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:25.450357 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" podStartSLOduration=1.450342904 podStartE2EDuration="1.450342904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:25.448929536 +0000 UTC m=+1583.791318630" watchObservedRunningTime="2026-11-02 19:55:25.450342904 +0000 UTC m=+1583.792731983" Nov 02 19:55:25.635616 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:25.631287 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:26.644761 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.644699 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:26.669904 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.669862 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:55:26.740851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.739996 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-certs\") pod \"5436015e-f85e-437c-b88c-7a30ddc02c77\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " Nov 02 19:55:26.740851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.740066 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-credential-socket\") pod \"5436015e-f85e-437c-b88c-7a30ddc02c77\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " Nov 02 19:55:26.740851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.740154 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-data\") pod \"5436015e-f85e-437c-b88c-7a30ddc02c77\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " Nov 02 19:55:26.740851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.740183 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-kube-api-access-7clpn\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-kube-api-access-7clpn\") pod \"5436015e-f85e-437c-b88c-7a30ddc02c77\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " Nov 02 19:55:26.740851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.740248 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-token\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-token\") pod \"5436015e-f85e-437c-b88c-7a30ddc02c77\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " Nov 02 19:55:26.740851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.740295 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5436015e-f85e-437c-b88c-7a30ddc02c77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5436015e-f85e-437c-b88c-7a30ddc02c77-istiod-ca-cert\") pod \"5436015e-f85e-437c-b88c-7a30ddc02c77\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " Nov 02 19:55:26.740851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.740444 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-podinfo\") pod \"5436015e-f85e-437c-b88c-7a30ddc02c77\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " Nov 02 19:55:26.741869 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.741683 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-data" pod "5436015e-f85e-437c-b88c-7a30ddc02c77" (UID: "5436015e-f85e-437c-b88c-7a30ddc02c77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.742348 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.742041 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-certs" pod "5436015e-f85e-437c-b88c-7a30ddc02c77" (UID: "5436015e-f85e-437c-b88c-7a30ddc02c77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.742988 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.742652 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-credential-socket" pod "5436015e-f85e-437c-b88c-7a30ddc02c77" (UID: "5436015e-f85e-437c-b88c-7a30ddc02c77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.743875 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.743838 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5436015e-f85e-437c-b88c-7a30ddc02c77-istiod-ca-cert" pod "5436015e-f85e-437c-b88c-7a30ddc02c77" (UID: "5436015e-f85e-437c-b88c-7a30ddc02c77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:26.745258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.744040 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-envoy\") pod \"5436015e-f85e-437c-b88c-7a30ddc02c77\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " Nov 02 19:55:26.745258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.744082 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-socket\") pod \"5436015e-f85e-437c-b88c-7a30ddc02c77\" (UID: \"5436015e-f85e-437c-b88c-7a30ddc02c77\") " Nov 02 19:55:26.745258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.744974 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.745258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.744996 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.745258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.745012 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.745258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.745024 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5436015e-f85e-437c-b88c-7a30ddc02c77-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.746626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.746585 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-socket" pod "5436015e-f85e-437c-b88c-7a30ddc02c77" (UID: "5436015e-f85e-437c-b88c-7a30ddc02c77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.749291 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.749243 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-kube-api-access-7clpn" pod "5436015e-f85e-437c-b88c-7a30ddc02c77" (UID: "5436015e-f85e-437c-b88c-7a30ddc02c77"). InnerVolumeSpecName "kube-api-access-7clpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:26.749291 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.749246 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-podinfo" pod "5436015e-f85e-437c-b88c-7a30ddc02c77" (UID: "5436015e-f85e-437c-b88c-7a30ddc02c77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:55:26.749711 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.749578 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-envoy" pod "5436015e-f85e-437c-b88c-7a30ddc02c77" (UID: "5436015e-f85e-437c-b88c-7a30ddc02c77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.751188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.751163 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-token" pod "5436015e-f85e-437c-b88c-7a30ddc02c77" (UID: "5436015e-f85e-437c-b88c-7a30ddc02c77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:26.849024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.848037 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7clpn\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-kube-api-access-7clpn\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.849024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.848142 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.849024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.848161 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.849024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.848176 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.849024 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:26.848191 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5436015e-f85e-437c-b88c-7a30ddc02c77-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:27.344915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.344870 5850 generic.go:356] "Generic (PLEG): container finished" podID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerID="f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a" exitCode=0 Nov 02 19:55:27.345074 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.344935 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" Nov 02 19:55:27.345074 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.344963 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" event={"ID":"5436015e-f85e-437c-b88c-7a30ddc02c77","Type":"ContainerDied","Data":"f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a"} Nov 02 19:55:27.345074 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.345018 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw" event={"ID":"5436015e-f85e-437c-b88c-7a30ddc02c77","Type":"ContainerDied","Data":"b4d163aefb1c04b2560153307f1604684a5f87a4b5e9855dd832d6273acb00af"} Nov 02 19:55:27.345074 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.345042 5850 scope.go:122] "RemoveContainer" containerID="f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a" Nov 02 19:55:27.345655 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.345600 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:27.346800 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.346779 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:55:27.357075 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.357053 5850 scope.go:122] "RemoveContainer" containerID="f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a" Nov 02 19:55:27.357398 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:55:27.357375 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a\": container with ID starting with f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a not found: ID does not exist" containerID="f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a" Nov 02 19:55:27.357513 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.357404 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a"} err="failed to get container status \"f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a\": rpc error: code = NotFound desc = could not find container \"f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a\": container with ID starting with f6832afc7e8bbd4f1221a0cab413fd9272f3727c03cd662749c9a7b3b752fe2a not found: ID does not exist" Nov 02 19:55:27.648654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.648616 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw"] Nov 02 19:55:27.652120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.652090 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-xtstw"] Nov 02 19:55:27.819105 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:27.819063 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" path="/var/lib/kubelet/pods/5436015e-f85e-437c-b88c-7a30ddc02c77/volumes" Nov 02 19:55:28.353193 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.353154 5850 generic.go:356] "Generic (PLEG): container finished" podID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerID="5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b" exitCode=0 Nov 02 19:55:28.353421 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.353226 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" event={"ID":"ed8dc527-83b5-48cd-8726-efe1912a67c9","Type":"ContainerDied","Data":"5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b"} Nov 02 19:55:28.353760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.353739 5850 scope.go:122] "RemoveContainer" containerID="5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b" Nov 02 19:55:28.355448 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.355413 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:55:28.355613 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.355517 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:55:28.355613 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.355573 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:55:28.617628 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.612246 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:28.778384 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.778317 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:28.820023 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.819984 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:55:28.951582 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:28.951494 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:29.338667 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:29.338574 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:29.471502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:29.464630 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" event={"ID":"ed8dc527-83b5-48cd-8726-efe1912a67c9","Type":"ContainerStarted","Data":"6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d"} Nov 02 19:55:30.177424 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:30.177384 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:55:30.178701 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:30.178673 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 19:55:30.178832 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:30.178730 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 19:55:31.176684 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:31.176649 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 19:55:31.176870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:31.176706 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 19:55:31.845733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:31.845694 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:32.189509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:32.187499 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 19:55:32.189509 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:32.187566 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 19:55:32.452551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:32.446166 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:33.177365 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:33.177325 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 19:55:33.177837 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:33.177427 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 19:55:33.256796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:33.256760 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585"] Nov 02 19:55:33.539669 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:33.539575 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:33.777361 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:33.777325 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:33.946998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:33.946964 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:34.103169 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.103081 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:34.176605 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.176573 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 19:55:34.176889 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.176859 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 19:55:34.336508 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.331029 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:34.851499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.850640 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2"] Nov 02 19:55:34.851499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.851080 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" containerID="cri-o://dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e" gracePeriod=30 Nov 02 19:55:34.928628 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.928593 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6"] Nov 02 19:55:34.929085 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.929056 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" containerID="cri-o://02c5de160dcc0ee0dd8580e6d2de7f870cf1d6d53132e5d6017bc3e2eabc04a9" gracePeriod=30 Nov 02 19:55:34.991573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.977582 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr"] Nov 02 19:55:34.991573 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:34.977859 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" containerID="cri-o://c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771" gracePeriod=30 Nov 02 19:55:35.044151 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:35.039550 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7"] Nov 02 19:55:35.044151 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:35.039816 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerName="istio-proxy" containerID="cri-o://adc5b08ab4e78fbaac501d6c7b901ec21a829a4645aebe989f88d625ae186920" gracePeriod=30 Nov 02 19:55:35.465227 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:35.463806 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:55:35.465227 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:35.465093 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:55:35.466901 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:35.466881 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 19:55:38.257772 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:38.257715 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" podUID="6444026e-0d2d-4cc7-923e-7013345cb966" containerName="payload-pre-processing" containerID="cri-o://ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78" gracePeriod=25 Nov 02 19:55:38.527216 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:38.527183 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:38.536642 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:38.536612 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:55:38.710085 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:38.710048 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6444026e-0d2d-4cc7-923e-7013345cb966-kube-api-access-z68zs\" (UniqueName: \"kubernetes.io/projected/6444026e-0d2d-4cc7-923e-7013345cb966-kube-api-access-z68zs\") pod \"6444026e-0d2d-4cc7-923e-7013345cb966\" (UID: \"6444026e-0d2d-4cc7-923e-7013345cb966\") " Nov 02 19:55:38.710444 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:38.710422 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6444026e-0d2d-4cc7-923e-7013345cb966-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6444026e-0d2d-4cc7-923e-7013345cb966-plugins-config-volume\") pod \"6444026e-0d2d-4cc7-923e-7013345cb966\" (UID: \"6444026e-0d2d-4cc7-923e-7013345cb966\") " Nov 02 19:55:38.710857 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:38.710831 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6444026e-0d2d-4cc7-923e-7013345cb966-plugins-config-volume" pod "6444026e-0d2d-4cc7-923e-7013345cb966" (UID: "6444026e-0d2d-4cc7-923e-7013345cb966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:38.711093 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:38.711077 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6444026e-0d2d-4cc7-923e-7013345cb966-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:38.712923 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:38.712893 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6444026e-0d2d-4cc7-923e-7013345cb966-kube-api-access-z68zs" pod "6444026e-0d2d-4cc7-923e-7013345cb966" (UID: "6444026e-0d2d-4cc7-923e-7013345cb966"). InnerVolumeSpecName "kube-api-access-z68zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:38.812633 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:38.812541 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68zs\" (UniqueName: \"kubernetes.io/projected/6444026e-0d2d-4cc7-923e-7013345cb966-kube-api-access-z68zs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:39.316803 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.316762 5850 generic.go:356] "Generic (PLEG): container finished" podID="6444026e-0d2d-4cc7-923e-7013345cb966" containerID="ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78" exitCode=0 Nov 02 19:55:39.317307 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.316824 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" Nov 02 19:55:39.317307 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.316823 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" event={"ID":"6444026e-0d2d-4cc7-923e-7013345cb966","Type":"ContainerDied","Data":"ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78"} Nov 02 19:55:39.317307 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.316981 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585" event={"ID":"6444026e-0d2d-4cc7-923e-7013345cb966","Type":"ContainerDied","Data":"7c50288bceb139e9e6f21a9253dbbfe47eb98ad23fc0c1fcfef555bc98916113"} Nov 02 19:55:39.317307 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.317020 5850 scope.go:122] "RemoveContainer" containerID="ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78" Nov 02 19:55:39.327918 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.327894 5850 scope.go:122] "RemoveContainer" containerID="ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78" Nov 02 19:55:39.328227 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:55:39.328207 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78\": container with ID starting with ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78 not found: ID does not exist" containerID="ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78" Nov 02 19:55:39.328267 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.328237 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78"} err="failed to get container status \"ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78\": rpc error: code = NotFound desc = could not find container \"ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78\": container with ID starting with ae296d5d36e95c51961c64a54d5fabd0ae642175395ca0d77715d683e156bd78 not found: ID does not exist" Nov 02 19:55:39.486273 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.486236 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585"] Nov 02 19:55:39.537885 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.537851 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4099cbd3-6dbbcb846c-v6585"] Nov 02 19:55:39.817204 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:39.817172 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6444026e-0d2d-4cc7-923e-7013345cb966" path="/var/lib/kubelet/pods/6444026e-0d2d-4cc7-923e-7013345cb966/volumes" Nov 02 19:55:40.245823 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.245796 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:55:40.282287 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.282258 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:55:40.329361 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.329324 5850 generic.go:356] "Generic (PLEG): container finished" podID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerID="02c5de160dcc0ee0dd8580e6d2de7f870cf1d6d53132e5d6017bc3e2eabc04a9" exitCode=0 Nov 02 19:55:40.329791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.329397 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" event={"ID":"e5aca617-dd63-4b17-8151-a864f0b23aab","Type":"ContainerDied","Data":"02c5de160dcc0ee0dd8580e6d2de7f870cf1d6d53132e5d6017bc3e2eabc04a9"} Nov 02 19:55:40.332376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.332346 5850 generic.go:356] "Generic (PLEG): container finished" podID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerID="adc5b08ab4e78fbaac501d6c7b901ec21a829a4645aebe989f88d625ae186920" exitCode=0 Nov 02 19:55:40.332570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.332424 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" event={"ID":"9144aefe-bea4-4000-bf7e-b2c016c2591e","Type":"ContainerDied","Data":"adc5b08ab4e78fbaac501d6c7b901ec21a829a4645aebe989f88d625ae186920"} Nov 02 19:55:40.333944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.333864 5850 generic.go:356] "Generic (PLEG): container finished" podID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerID="dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e" exitCode=0 Nov 02 19:55:40.333944 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.333910 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" Nov 02 19:55:40.334121 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.333962 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" event={"ID":"dd6a4b92-1091-419e-ba15-1a6af5381698","Type":"ContainerDied","Data":"dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e"} Nov 02 19:55:40.334121 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.334034 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2" event={"ID":"dd6a4b92-1091-419e-ba15-1a6af5381698","Type":"ContainerDied","Data":"612ccb6fc10c2969394e203edea87e2d4fa266a164484dc5ee55e22bea53aa33"} Nov 02 19:55:40.334121 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.334066 5850 scope.go:122] "RemoveContainer" containerID="dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e" Nov 02 19:55:40.335642 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.335609 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-token\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-token\") pod \"dd6a4b92-1091-419e-ba15-1a6af5381698\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " Nov 02 19:55:40.335739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.335648 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-credential-socket\") pod \"dd6a4b92-1091-419e-ba15-1a6af5381698\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " Nov 02 19:55:40.335739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.335688 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-kube-api-access-hlkk2\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-kube-api-access-hlkk2\") pod \"dd6a4b92-1091-419e-ba15-1a6af5381698\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " Nov 02 19:55:40.335851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.335757 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6a4b92-1091-419e-ba15-1a6af5381698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6a4b92-1091-419e-ba15-1a6af5381698-istiod-ca-cert\") pod \"dd6a4b92-1091-419e-ba15-1a6af5381698\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " Nov 02 19:55:40.335851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.335790 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-data\") pod \"dd6a4b92-1091-419e-ba15-1a6af5381698\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " Nov 02 19:55:40.335851 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.335824 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-socket\") pod \"dd6a4b92-1091-419e-ba15-1a6af5381698\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " Nov 02 19:55:40.336003 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.335963 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-podinfo\") pod \"dd6a4b92-1091-419e-ba15-1a6af5381698\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " Nov 02 19:55:40.336062 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.336023 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-envoy\") pod \"dd6a4b92-1091-419e-ba15-1a6af5381698\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " Nov 02 19:55:40.336121 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.336068 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-certs\") pod \"dd6a4b92-1091-419e-ba15-1a6af5381698\" (UID: \"dd6a4b92-1091-419e-ba15-1a6af5381698\") " Nov 02 19:55:40.336843 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.336820 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-certs" pod "dd6a4b92-1091-419e-ba15-1a6af5381698" (UID: "dd6a4b92-1091-419e-ba15-1a6af5381698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.337318 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.337249 5850 generic.go:356] "Generic (PLEG): container finished" podID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerID="c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771" exitCode=0 Nov 02 19:55:40.337424 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.337327 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-data" pod "dd6a4b92-1091-419e-ba15-1a6af5381698" (UID: "dd6a4b92-1091-419e-ba15-1a6af5381698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.337424 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.337393 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" event={"ID":"71596876-6e7f-47d5-8972-ea460f0fa1f6","Type":"ContainerDied","Data":"c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771"} Nov 02 19:55:40.337597 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.337439 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" event={"ID":"71596876-6e7f-47d5-8972-ea460f0fa1f6","Type":"ContainerDied","Data":"a0be621d23e63791aa97c8791b6fbad07e37bb365df84e03d22399cc166666cb"} Nov 02 19:55:40.337597 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.337569 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr" Nov 02 19:55:40.337597 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.337576 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-credential-socket" pod "dd6a4b92-1091-419e-ba15-1a6af5381698" (UID: "dd6a4b92-1091-419e-ba15-1a6af5381698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.338441 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.338414 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6a4b92-1091-419e-ba15-1a6af5381698-istiod-ca-cert" pod "dd6a4b92-1091-419e-ba15-1a6af5381698" (UID: "dd6a4b92-1091-419e-ba15-1a6af5381698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:40.338687 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.338641 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-socket" pod "dd6a4b92-1091-419e-ba15-1a6af5381698" (UID: "dd6a4b92-1091-419e-ba15-1a6af5381698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.340547 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.340513 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-kube-api-access-hlkk2" pod "dd6a4b92-1091-419e-ba15-1a6af5381698" (UID: "dd6a4b92-1091-419e-ba15-1a6af5381698"). InnerVolumeSpecName "kube-api-access-hlkk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:40.341184 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.341155 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-envoy" pod "dd6a4b92-1091-419e-ba15-1a6af5381698" (UID: "dd6a4b92-1091-419e-ba15-1a6af5381698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.341339 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.341258 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-token" pod "dd6a4b92-1091-419e-ba15-1a6af5381698" (UID: "dd6a4b92-1091-419e-ba15-1a6af5381698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:40.341747 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.341723 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-podinfo" pod "dd6a4b92-1091-419e-ba15-1a6af5381698" (UID: "dd6a4b92-1091-419e-ba15-1a6af5381698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:55:40.346525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.346467 5850 scope.go:122] "RemoveContainer" containerID="dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e" Nov 02 19:55:40.346897 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:55:40.346830 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e\": container with ID starting with dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e not found: ID does not exist" containerID="dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e" Nov 02 19:55:40.346897 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.346868 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e"} err="failed to get container status \"dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e\": rpc error: code = NotFound desc = could not find container \"dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e\": container with ID starting with dc6ac207cbe9e4271197b6493c1efd658139a6fe54ff75cd86f51b36c21e645e not found: ID does not exist" Nov 02 19:55:40.346897 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.346894 5850 scope.go:122] "RemoveContainer" containerID="c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771" Nov 02 19:55:40.357396 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.357369 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:55:40.359117 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.359092 5850 scope.go:122] "RemoveContainer" containerID="c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771" Nov 02 19:55:40.359419 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:55:40.359387 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771\": container with ID starting with c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771 not found: ID does not exist" containerID="c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771" Nov 02 19:55:40.359570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.359430 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771"} err="failed to get container status \"c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771\": rpc error: code = NotFound desc = could not find container \"c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771\": container with ID starting with c8eb75be277b8af28e44976e8909ef02c7136271ba378ce0d4c8856c30431771 not found: ID does not exist" Nov 02 19:55:40.361802 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.361783 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:55:40.437489 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437432 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-credential-socket\") pod \"71596876-6e7f-47d5-8972-ea460f0fa1f6\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " Nov 02 19:55:40.437742 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437527 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-kube-api-access-qz2vp\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-kube-api-access-qz2vp\") pod \"e5aca617-dd63-4b17-8151-a864f0b23aab\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " Nov 02 19:55:40.437742 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437646 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-podinfo\") pod \"71596876-6e7f-47d5-8972-ea460f0fa1f6\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " Nov 02 19:55:40.437742 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437722 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-envoy\") pod \"e5aca617-dd63-4b17-8151-a864f0b23aab\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " Nov 02 19:55:40.437922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437793 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5aca617-dd63-4b17-8151-a864f0b23aab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aca617-dd63-4b17-8151-a864f0b23aab-istiod-ca-cert\") pod \"e5aca617-dd63-4b17-8151-a864f0b23aab\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " Nov 02 19:55:40.437922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437789 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-credential-socket" pod "71596876-6e7f-47d5-8972-ea460f0fa1f6" (UID: "71596876-6e7f-47d5-8972-ea460f0fa1f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.437922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437825 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-data\") pod \"e5aca617-dd63-4b17-8151-a864f0b23aab\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " Nov 02 19:55:40.437922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437866 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-credential-socket\") pod \"e5aca617-dd63-4b17-8151-a864f0b23aab\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " Nov 02 19:55:40.437922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437899 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-kube-api-access-pqmjq\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-kube-api-access-pqmjq\") pod \"71596876-6e7f-47d5-8972-ea460f0fa1f6\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " Nov 02 19:55:40.438183 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.437976 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71596876-6e7f-47d5-8972-ea460f0fa1f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71596876-6e7f-47d5-8972-ea460f0fa1f6-istiod-ca-cert\") pod \"71596876-6e7f-47d5-8972-ea460f0fa1f6\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " Nov 02 19:55:40.438183 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438040 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-token\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-token\") pod \"71596876-6e7f-47d5-8972-ea460f0fa1f6\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " Nov 02 19:55:40.438183 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438075 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-socket\") pod \"e5aca617-dd63-4b17-8151-a864f0b23aab\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " Nov 02 19:55:40.438183 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438151 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-certs\") pod \"71596876-6e7f-47d5-8972-ea460f0fa1f6\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " Nov 02 19:55:40.438382 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438198 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-envoy\") pod \"71596876-6e7f-47d5-8972-ea460f0fa1f6\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " Nov 02 19:55:40.438382 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438227 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-socket\") pod \"71596876-6e7f-47d5-8972-ea460f0fa1f6\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " Nov 02 19:55:40.438382 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438259 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-podinfo\") pod \"e5aca617-dd63-4b17-8151-a864f0b23aab\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " Nov 02 19:55:40.438382 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438375 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-certs\") pod \"e5aca617-dd63-4b17-8151-a864f0b23aab\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " Nov 02 19:55:40.438664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438408 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-token\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-token\") pod \"e5aca617-dd63-4b17-8151-a864f0b23aab\" (UID: \"e5aca617-dd63-4b17-8151-a864f0b23aab\") " Nov 02 19:55:40.438664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438455 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-data\") pod \"71596876-6e7f-47d5-8972-ea460f0fa1f6\" (UID: \"71596876-6e7f-47d5-8972-ea460f0fa1f6\") " Nov 02 19:55:40.438838 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.438811 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-socket" pod "e5aca617-dd63-4b17-8151-a864f0b23aab" (UID: "e5aca617-dd63-4b17-8151-a864f0b23aab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439205 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-credential-socket" pod "e5aca617-dd63-4b17-8151-a864f0b23aab" (UID: "e5aca617-dd63-4b17-8151-a864f0b23aab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439409 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439432 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439447 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439462 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlkk2\" (UniqueName: \"kubernetes.io/projected/dd6a4b92-1091-419e-ba15-1a6af5381698-kube-api-access-hlkk2\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439516 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6a4b92-1091-419e-ba15-1a6af5381698-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439532 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439545 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439559 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439573 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439588 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439603 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439616 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6a4b92-1091-419e-ba15-1a6af5381698-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439612 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-certs" pod "71596876-6e7f-47d5-8972-ea460f0fa1f6" (UID: "71596876-6e7f-47d5-8972-ea460f0fa1f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439717 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-socket" pod "71596876-6e7f-47d5-8972-ea460f0fa1f6" (UID: "71596876-6e7f-47d5-8972-ea460f0fa1f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439847 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-data" pod "e5aca617-dd63-4b17-8151-a864f0b23aab" (UID: "e5aca617-dd63-4b17-8151-a864f0b23aab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.440008 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.439978 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5aca617-dd63-4b17-8151-a864f0b23aab-istiod-ca-cert" pod "e5aca617-dd63-4b17-8151-a864f0b23aab" (UID: "e5aca617-dd63-4b17-8151-a864f0b23aab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:40.441495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.440238 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-data" pod "71596876-6e7f-47d5-8972-ea460f0fa1f6" (UID: "71596876-6e7f-47d5-8972-ea460f0fa1f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.441495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.440256 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71596876-6e7f-47d5-8972-ea460f0fa1f6-istiod-ca-cert" pod "71596876-6e7f-47d5-8972-ea460f0fa1f6" (UID: "71596876-6e7f-47d5-8972-ea460f0fa1f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:40.441495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.440328 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-certs" pod "e5aca617-dd63-4b17-8151-a864f0b23aab" (UID: "e5aca617-dd63-4b17-8151-a864f0b23aab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.441495 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.440633 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-kube-api-access-qz2vp" pod "e5aca617-dd63-4b17-8151-a864f0b23aab" (UID: "e5aca617-dd63-4b17-8151-a864f0b23aab"). InnerVolumeSpecName "kube-api-access-qz2vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:40.442559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.442441 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-token" pod "71596876-6e7f-47d5-8972-ea460f0fa1f6" (UID: "71596876-6e7f-47d5-8972-ea460f0fa1f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:40.442559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.442524 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-token" pod "e5aca617-dd63-4b17-8151-a864f0b23aab" (UID: "e5aca617-dd63-4b17-8151-a864f0b23aab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:40.442964 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.442906 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-envoy" pod "e5aca617-dd63-4b17-8151-a864f0b23aab" (UID: "e5aca617-dd63-4b17-8151-a864f0b23aab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.442964 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.442925 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-podinfo" pod "71596876-6e7f-47d5-8972-ea460f0fa1f6" (UID: "71596876-6e7f-47d5-8972-ea460f0fa1f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:55:40.443557 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.443534 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-envoy" pod "71596876-6e7f-47d5-8972-ea460f0fa1f6" (UID: "71596876-6e7f-47d5-8972-ea460f0fa1f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.444730 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.444703 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-podinfo" pod "e5aca617-dd63-4b17-8151-a864f0b23aab" (UID: "e5aca617-dd63-4b17-8151-a864f0b23aab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:55:40.445537 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.445467 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-kube-api-access-pqmjq" pod "71596876-6e7f-47d5-8972-ea460f0fa1f6" (UID: "71596876-6e7f-47d5-8972-ea460f0fa1f6"). InnerVolumeSpecName "kube-api-access-pqmjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:40.540982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.540946 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-certs\") pod \"9144aefe-bea4-4000-bf7e-b2c016c2591e\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " Nov 02 19:55:40.541173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.541040 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-token\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-token\") pod \"9144aefe-bea4-4000-bf7e-b2c016c2591e\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " Nov 02 19:55:40.541173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.541097 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-credential-socket\") pod \"9144aefe-bea4-4000-bf7e-b2c016c2591e\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " Nov 02 19:55:40.541294 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.541182 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-data\") pod \"9144aefe-bea4-4000-bf7e-b2c016c2591e\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " Nov 02 19:55:40.541294 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.541206 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-kube-api-access-vfscd\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-kube-api-access-vfscd\") pod \"9144aefe-bea4-4000-bf7e-b2c016c2591e\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " Nov 02 19:55:40.541294 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.541236 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-podinfo\") pod \"9144aefe-bea4-4000-bf7e-b2c016c2591e\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " Nov 02 19:55:40.541581 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.541469 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-credential-socket" pod "9144aefe-bea4-4000-bf7e-b2c016c2591e" (UID: "9144aefe-bea4-4000-bf7e-b2c016c2591e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.541833 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.541807 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-socket\") pod \"9144aefe-bea4-4000-bf7e-b2c016c2591e\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " Nov 02 19:55:40.542002 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.541925 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-envoy\") pod \"9144aefe-bea4-4000-bf7e-b2c016c2591e\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " Nov 02 19:55:40.542127 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542106 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9144aefe-bea4-4000-bf7e-b2c016c2591e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9144aefe-bea4-4000-bf7e-b2c016c2591e-istiod-ca-cert\") pod \"9144aefe-bea4-4000-bf7e-b2c016c2591e\" (UID: \"9144aefe-bea4-4000-bf7e-b2c016c2591e\") " Nov 02 19:55:40.542363 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.541929 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-data" pod "9144aefe-bea4-4000-bf7e-b2c016c2591e" (UID: "9144aefe-bea4-4000-bf7e-b2c016c2591e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.542455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542107 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-socket" pod "9144aefe-bea4-4000-bf7e-b2c016c2591e" (UID: "9144aefe-bea4-4000-bf7e-b2c016c2591e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.542455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542246 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-certs" pod "9144aefe-bea4-4000-bf7e-b2c016c2591e" (UID: "9144aefe-bea4-4000-bf7e-b2c016c2591e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.542600 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542528 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9144aefe-bea4-4000-bf7e-b2c016c2591e-istiod-ca-cert" pod "9144aefe-bea4-4000-bf7e-b2c016c2591e" (UID: "9144aefe-bea4-4000-bf7e-b2c016c2591e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:40.542937 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542911 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz2vp\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-kube-api-access-qz2vp\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542939 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542953 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542967 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542978 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.542990 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aca617-dd63-4b17-8151-a864f0b23aab-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543002 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543022 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543013 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqmjq\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-kube-api-access-pqmjq\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543029 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9144aefe-bea4-4000-bf7e-b2c016c2591e-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543040 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71596876-6e7f-47d5-8972-ea460f0fa1f6-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543052 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543065 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543077 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543088 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543143 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543157 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543169 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aca617-dd63-4b17-8151-a864f0b23aab-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543182 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5aca617-dd63-4b17-8151-a864f0b23aab-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543195 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.543697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.543208 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71596876-6e7f-47d5-8972-ea460f0fa1f6-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.544634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.544609 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-kube-api-access-vfscd" pod "9144aefe-bea4-4000-bf7e-b2c016c2591e" (UID: "9144aefe-bea4-4000-bf7e-b2c016c2591e"). InnerVolumeSpecName "kube-api-access-vfscd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:40.544634 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.544622 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-envoy" pod "9144aefe-bea4-4000-bf7e-b2c016c2591e" (UID: "9144aefe-bea4-4000-bf7e-b2c016c2591e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:40.544733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.544691 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-token" pod "9144aefe-bea4-4000-bf7e-b2c016c2591e" (UID: "9144aefe-bea4-4000-bf7e-b2c016c2591e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:40.545633 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.545617 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-podinfo" pod "9144aefe-bea4-4000-bf7e-b2c016c2591e" (UID: "9144aefe-bea4-4000-bf7e-b2c016c2591e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:55:40.663322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.660811 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfscd\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-kube-api-access-vfscd\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.663322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.660849 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.663322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.660874 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.663322 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.660889 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9144aefe-bea4-4000-bf7e-b2c016c2591e-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:40.965410 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.965315 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2"] Nov 02 19:55:40.968817 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:40.968786 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-b6sf2"] Nov 02 19:55:41.079462 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.079427 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr"] Nov 02 19:55:41.083654 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.083625 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-j2krr"] Nov 02 19:55:41.345652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.345567 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" Nov 02 19:55:41.346123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.345570 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6" event={"ID":"e5aca617-dd63-4b17-8151-a864f0b23aab","Type":"ContainerDied","Data":"405b0e674d78ff1ad3e903295107606585c0bf5d6fe2eabff9bd0c559b0aa138"} Nov 02 19:55:41.346123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.345721 5850 scope.go:122] "RemoveContainer" containerID="02c5de160dcc0ee0dd8580e6d2de7f870cf1d6d53132e5d6017bc3e2eabc04a9" Nov 02 19:55:41.347127 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.347092 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" Nov 02 19:55:41.347127 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.347112 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7" event={"ID":"9144aefe-bea4-4000-bf7e-b2c016c2591e","Type":"ContainerDied","Data":"bd7c7d0f8d75eaeb74a047cc7d495f31e1e1fea6f7997b54faf5fe020f90b3e8"} Nov 02 19:55:41.359515 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.359461 5850 scope.go:122] "RemoveContainer" containerID="adc5b08ab4e78fbaac501d6c7b901ec21a829a4645aebe989f88d625ae186920" Nov 02 19:55:41.514592 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.514551 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7"] Nov 02 19:55:41.519029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.518998 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qvgf7"] Nov 02 19:55:41.631055 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.631017 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6"] Nov 02 19:55:41.634290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.634256 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vsrx6"] Nov 02 19:55:41.816346 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.816310 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" path="/var/lib/kubelet/pods/71596876-6e7f-47d5-8972-ea460f0fa1f6/volumes" Nov 02 19:55:41.816911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.816895 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" path="/var/lib/kubelet/pods/9144aefe-bea4-4000-bf7e-b2c016c2591e/volumes" Nov 02 19:55:41.817337 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.817325 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" path="/var/lib/kubelet/pods/dd6a4b92-1091-419e-ba15-1a6af5381698/volumes" Nov 02 19:55:41.818214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:41.818194 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" path="/var/lib/kubelet/pods/e5aca617-dd63-4b17-8151-a864f0b23aab/volumes" Nov 02 19:55:42.927264 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:42.927231 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:44.762892 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:55:44.762849 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:55:47.335434 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:47.335399 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:51.736440 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:51.736403 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:56.336788 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:56.336754 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:57.325352 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:57.325317 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:59.004827 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:59.004794 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:59.167237 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:59.167202 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:59.576762 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:55:59.576724 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:00.054592 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.054543 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g"] Nov 02 19:56:00.055572 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055544 5850 cpu_manager.go:414] "containerMap: removing container" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" Nov 02 19:56:00.055572 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055568 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055591 5850 cpu_manager.go:414] "containerMap: removing container" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055597 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055609 5850 cpu_manager.go:414] "containerMap: removing container" podUID="6444026e-0d2d-4cc7-923e-7013345cb966" containerName="payload-pre-processing" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055616 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6444026e-0d2d-4cc7-923e-7013345cb966" containerName="payload-pre-processing" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055627 5850 cpu_manager.go:414] "containerMap: removing container" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerName="istio-proxy" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055632 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerName="istio-proxy" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055646 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055652 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055688 5850 cpu_manager.go:414] "containerMap: removing container" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" Nov 02 19:56:00.055739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055695 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" Nov 02 19:56:00.056088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055854 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="5436015e-f85e-437c-b88c-7a30ddc02c77" containerName="istio-proxy" Nov 02 19:56:00.056088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055865 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6a4b92-1091-419e-ba15-1a6af5381698" containerName="istio-proxy" Nov 02 19:56:00.056088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055874 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="6444026e-0d2d-4cc7-923e-7013345cb966" containerName="payload-pre-processing" Nov 02 19:56:00.056088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055883 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="9144aefe-bea4-4000-bf7e-b2c016c2591e" containerName="istio-proxy" Nov 02 19:56:00.056088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055891 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5aca617-dd63-4b17-8151-a864f0b23aab" containerName="istio-proxy" Nov 02 19:56:00.056088 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.055901 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="71596876-6e7f-47d5-8972-ea460f0fa1f6" containerName="istio-proxy" Nov 02 19:56:00.059896 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.059875 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.119630 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.114852 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0b7eed-openshift-default-dockercfg-6lw2q\"" Nov 02 19:56:00.183553 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.168533 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-envoy\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.183940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.183905 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-socket\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.184108 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.184094 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-certs\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.184257 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.184244 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-data\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.187502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.184385 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-credential-socket\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.187799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.187770 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69514c97-be51-45d0-818f-29c4d2e69318-istiod-ca-cert\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.191581 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.191551 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-istio-token\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.199503 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.193395 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69514c97-be51-45d0-818f-29c4d2e69318-istio-podinfo\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.209506 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.199846 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjdpq\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-kube-api-access-qjdpq\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.368014 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.368772 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69514c97-be51-45d0-818f-29c4d2e69318-istiod-ca-cert\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.368822 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-istio-token\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.368888 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69514c97-be51-45d0-818f-29c4d2e69318-istio-podinfo\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.368980 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjdpq\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-kube-api-access-qjdpq\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.369025 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-envoy\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.369115 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-socket\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.369168 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-certs\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.369233 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-data\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.369308 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-credential-socket\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.370292 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69514c97-be51-45d0-818f-29c4d2e69318-istiod-ca-cert\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.371156 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-credential-socket\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.371392 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-socket\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.371995 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-certs\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.374584 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.372219 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-data\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.384588 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.384548 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-envoy\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.396150 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.396090 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69514c97-be51-45d0-818f-29c4d2e69318-istio-podinfo\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.442876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.442767 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g"] Nov 02 19:56:00.460410 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.460328 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjdpq\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-kube-api-access-qjdpq\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.481402 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.469035 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-istio-token\") pod \"e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.524065 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.524031 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:00.674608 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.674505 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:00.935727 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:00.935648 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:01.100802 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.100760 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g"] Nov 02 19:56:01.135146 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:56:01.135102 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69514c97_be51_45d0_818f_29c4d2e69318.slice/crio-15b118ffd20bd3afc334a2630645953e4b071bbd1ef5f9db79275049b4305d69 WatchSource:0}: Error finding container 15b118ffd20bd3afc334a2630645953e4b071bbd1ef5f9db79275049b4305d69: Status 404 returned error can't find the container with id 15b118ffd20bd3afc334a2630645953e4b071bbd1ef5f9db79275049b4305d69 Nov 02 19:56:01.137697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.137661 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:01.137805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.137735 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:01.137805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.137767 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:01.335121 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.335081 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:01.412661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.412615 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l"] Nov 02 19:56:01.417061 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.417031 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.516544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.512855 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vc9p\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-kube-api-access-7vc9p\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.516544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.512901 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-certs\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.516544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.512947 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-credential-socket\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.516544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.512999 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-istio-token\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.516544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.513074 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa77d63a-7ab5-4e82-9144-15039098659b-istio-podinfo\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.516544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.513114 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-envoy\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.516544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.513142 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-data\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.516544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.513176 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-socket\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.516544 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.513255 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa77d63a-7ab5-4e82-9144-15039098659b-istiod-ca-cert\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.530295 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.529789 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-89a22a-openshift-default-dockercfg-xpdwq\"" Nov 02 19:56:01.531945 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.531312 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:01.632934 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.600549 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l"] Nov 02 19:56:01.760032 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.759991 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-istio-token\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.761806 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.761763 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa77d63a-7ab5-4e82-9144-15039098659b-istio-podinfo\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.761982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.761839 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-envoy\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.761982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.761877 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-data\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.761982 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.761922 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-socket\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.762168 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.762031 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa77d63a-7ab5-4e82-9144-15039098659b-istiod-ca-cert\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.762231 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.762206 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vc9p\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-kube-api-access-7vc9p\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.762297 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.762246 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-certs\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.762350 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.762312 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-credential-socket\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.762843 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.762779 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-credential-socket\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.763070 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.763044 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-data\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.763774 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.763608 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa77d63a-7ab5-4e82-9144-15039098659b-istiod-ca-cert\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.763774 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.763613 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-certs\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.763933 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.763897 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-socket\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.771870 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.771780 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" event={"ID":"69514c97-be51-45d0-818f-29c4d2e69318","Type":"ContainerStarted","Data":"4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d"} Nov 02 19:56:01.772143 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.772118 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" event={"ID":"69514c97-be51-45d0-818f-29c4d2e69318","Type":"ContainerStarted","Data":"15b118ffd20bd3afc334a2630645953e4b071bbd1ef5f9db79275049b4305d69"} Nov 02 19:56:01.804798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.791259 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa77d63a-7ab5-4e82-9144-15039098659b-istio-podinfo\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.806533 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.805517 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-envoy\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.852037 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.852002 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vc9p\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-kube-api-access-7vc9p\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.868546 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.860400 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-istio-token\") pod \"e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:01.925360 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.925287 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" podStartSLOduration=1.925266854 podStartE2EDuration="1.925266854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:01.923294455 +0000 UTC m=+1620.265683561" watchObservedRunningTime="2026-11-02 19:56:01.925266854 +0000 UTC m=+1620.267655936" Nov 02 19:56:01.954977 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:01.954931 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:02.094618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.092728 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-89a22a-openshift-default-dockercfg-xpdwq\"" Nov 02 19:56:02.099652 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.098983 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:02.515455 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.515413 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:02.635067 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:56:02.635029 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa77d63a_7ab5_4e82_9144_15039098659b.slice/crio-d0ee92fa1e08dd1d6aec2acac319da66bf1182e9894c48a22829d318b9b6b0e6 WatchSource:0}: Error finding container d0ee92fa1e08dd1d6aec2acac319da66bf1182e9894c48a22829d318b9b6b0e6: Status 404 returned error can't find the container with id d0ee92fa1e08dd1d6aec2acac319da66bf1182e9894c48a22829d318b9b6b0e6 Nov 02 19:56:02.637958 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.637937 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l"] Nov 02 19:56:02.638428 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.638389 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:02.638514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.638489 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:02.638581 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.638537 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:02.675633 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.675601 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:02.678315 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.678277 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:02.681852 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.681829 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:02.781505 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.781394 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" event={"ID":"aa77d63a-7ab5-4e82-9144-15039098659b","Type":"ContainerStarted","Data":"d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc"} Nov 02 19:56:02.781505 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.781454 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" event={"ID":"aa77d63a-7ab5-4e82-9144-15039098659b","Type":"ContainerStarted","Data":"d0ee92fa1e08dd1d6aec2acac319da66bf1182e9894c48a22829d318b9b6b0e6"} Nov 02 19:56:02.781754 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.781693 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:02.783013 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.782990 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:56:02.922535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:02.922455 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" podStartSLOduration=1.922434586 podStartE2EDuration="1.922434586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:02.919985311 +0000 UTC m=+1621.262374391" watchObservedRunningTime="2026-11-02 19:56:02.922434586 +0000 UTC m=+1621.264823670" Nov 02 19:56:03.100672 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:03.100350 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:03.111258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:03.110626 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 19:56:03.111258 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:03.110693 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 19:56:04.104822 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:04.104790 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:04.713060 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:04.713027 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:04.935615 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:04.935078 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:04.935615 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:04.935151 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:04.959499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:04.955623 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:56:05.109784 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:05.109657 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:05.706551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:05.693357 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:05.879725 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:05.879681 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg"] Nov 02 19:56:05.880197 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:05.880148 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" podUID="da5f54e7-06b6-4e0f-8d71-b24aae87c630" containerName="maas-api" containerID="cri-o://0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d" gracePeriod=30 Nov 02 19:56:05.902173 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:05.902148 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:06.086499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.077264 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:06.643546 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.642772 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:56:06.736974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.736934 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-metrics-certs\") pod \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " Nov 02 19:56:06.737225 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.737207 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5f54e7-06b6-4e0f-8d71-b24aae87c630-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5f54e7-06b6-4e0f-8d71-b24aae87c630-openshift-service-ca\") pod \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " Nov 02 19:56:06.737320 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.737303 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5f54e7-06b6-4e0f-8d71-b24aae87c630-kube-api-access-7x6zg\" (UniqueName: \"kubernetes.io/projected/da5f54e7-06b6-4e0f-8d71-b24aae87c630-kube-api-access-7x6zg\") pod \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " Nov 02 19:56:06.737381 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.737362 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls\") pod \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\" (UID: \"da5f54e7-06b6-4e0f-8d71-b24aae87c630\") " Nov 02 19:56:06.737700 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.737664 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5f54e7-06b6-4e0f-8d71-b24aae87c630-openshift-service-ca" pod "da5f54e7-06b6-4e0f-8d71-b24aae87c630" (UID: "da5f54e7-06b6-4e0f-8d71-b24aae87c630"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:56:06.738153 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.738123 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5f54e7-06b6-4e0f-8d71-b24aae87c630-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:06.739295 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.739269 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-metrics-certs" pod "da5f54e7-06b6-4e0f-8d71-b24aae87c630" (UID: "da5f54e7-06b6-4e0f-8d71-b24aae87c630"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:56:06.739946 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.739913 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls" pod "da5f54e7-06b6-4e0f-8d71-b24aae87c630" (UID: "da5f54e7-06b6-4e0f-8d71-b24aae87c630"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:56:06.740049 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.739919 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5f54e7-06b6-4e0f-8d71-b24aae87c630-kube-api-access-7x6zg" pod "da5f54e7-06b6-4e0f-8d71-b24aae87c630" (UID: "da5f54e7-06b6-4e0f-8d71-b24aae87c630"). InnerVolumeSpecName "kube-api-access-7x6zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:06.839880 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.839789 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x6zg\" (UniqueName: \"kubernetes.io/projected/da5f54e7-06b6-4e0f-8d71-b24aae87c630-kube-api-access-7x6zg\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:06.839880 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.839833 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:06.839880 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.839844 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5f54e7-06b6-4e0f-8d71-b24aae87c630-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:06.955649 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.955611 5850 generic.go:356] "Generic (PLEG): container finished" podID="da5f54e7-06b6-4e0f-8d71-b24aae87c630" containerID="0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d" exitCode=0 Nov 02 19:56:06.955840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.955668 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" Nov 02 19:56:06.955840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.955689 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" event={"ID":"da5f54e7-06b6-4e0f-8d71-b24aae87c630","Type":"ContainerDied","Data":"0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d"} Nov 02 19:56:06.955840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.955747 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg" event={"ID":"da5f54e7-06b6-4e0f-8d71-b24aae87c630","Type":"ContainerDied","Data":"9fd19ac3b30d36e4d817b496f983f707edbbf1fd70a74c39c8d177bbe79c4bec"} Nov 02 19:56:06.955840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.955775 5850 scope.go:122] "RemoveContainer" containerID="0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d" Nov 02 19:56:06.967798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.967776 5850 scope.go:122] "RemoveContainer" containerID="0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d" Nov 02 19:56:06.968107 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:56:06.968089 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d\": container with ID starting with 0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d not found: ID does not exist" containerID="0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d" Nov 02 19:56:06.968175 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:06.968118 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d"} err="failed to get container status \"0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d\": rpc error: code = NotFound desc = could not find container \"0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d\": container with ID starting with 0e47e9c4ec6de0a8c1c8c8526678ccf5be88ceed7b40583daab55f33d2612c8d not found: ID does not exist" Nov 02 19:56:07.220514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:07.217713 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg"] Nov 02 19:56:07.272498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:07.272416 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4099cbd3-5cdc475576-9xsjg"] Nov 02 19:56:07.837357 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:07.837318 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da5f54e7-06b6-4e0f-8d71-b24aae87c630" path="/var/lib/kubelet/pods/da5f54e7-06b6-4e0f-8d71-b24aae87c630/volumes" Nov 02 19:56:08.128499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.127555 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:08.289290 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.289253 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:08.494105 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.494011 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:08.817524 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.817418 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:08.832425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.832377 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn"] Nov 02 19:56:08.833832 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.833797 5850 cpu_manager.go:414] "containerMap: removing container" podUID="da5f54e7-06b6-4e0f-8d71-b24aae87c630" containerName="maas-api" Nov 02 19:56:08.833998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.833983 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5f54e7-06b6-4e0f-8d71-b24aae87c630" containerName="maas-api" Nov 02 19:56:08.834296 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.834277 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5f54e7-06b6-4e0f-8d71-b24aae87c630" containerName="maas-api" Nov 02 19:56:08.840296 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.840271 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:08.956223 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.943448 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0b7eed-dockercfg-xrkhw\"" Nov 02 19:56:08.981953 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:08.958116 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0b7eed\"" Nov 02 19:56:09.030905 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.030863 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0005d91-7470-4ec7-9087-a53710e3e94c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn\" (UID: \"b0005d91-7470-4ec7-9087-a53710e3e94c\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:09.031533 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.031500 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz46k\" (UniqueName: \"kubernetes.io/projected/b0005d91-7470-4ec7-9087-a53710e3e94c-kube-api-access-mz46k\") pod \"payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn\" (UID: \"b0005d91-7470-4ec7-9087-a53710e3e94c\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:09.189611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.170140 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:09.189611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.173314 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn"] Nov 02 19:56:09.265098 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.261387 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0005d91-7470-4ec7-9087-a53710e3e94c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn\" (UID: \"b0005d91-7470-4ec7-9087-a53710e3e94c\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:09.265098 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.261733 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz46k\" (UniqueName: \"kubernetes.io/projected/b0005d91-7470-4ec7-9087-a53710e3e94c-kube-api-access-mz46k\") pod \"payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn\" (UID: \"b0005d91-7470-4ec7-9087-a53710e3e94c\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:09.265098 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.263516 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0005d91-7470-4ec7-9087-a53710e3e94c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn\" (UID: \"b0005d91-7470-4ec7-9087-a53710e3e94c\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:09.272238 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.272206 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp"] Nov 02 19:56:09.277242 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.277217 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp"] Nov 02 19:56:09.277370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.277301 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:09.333831 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.333758 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz46k\" (UniqueName: \"kubernetes.io/projected/b0005d91-7470-4ec7-9087-a53710e3e94c-kube-api-access-mz46k\") pod \"payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn\" (UID: \"b0005d91-7470-4ec7-9087-a53710e3e94c\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:09.454906 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.454812 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:09.464813 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.464777 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp\" (UID: \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:09.464964 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.464890 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4khk7\" (UniqueName: \"kubernetes.io/projected/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-kube-api-access-4khk7\") pod \"payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp\" (UID: \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:09.566699 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.566656 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp\" (UID: \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:09.566876 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.566739 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4khk7\" (UniqueName: \"kubernetes.io/projected/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-kube-api-access-4khk7\") pod \"payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp\" (UID: \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:09.567904 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.567880 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp\" (UID: \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:09.592626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.592587 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:09.635254 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.635208 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4khk7\" (UniqueName: \"kubernetes.io/projected/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-kube-api-access-4khk7\") pod \"payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp\" (UID: \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:09.852688 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.852608 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:09.863611 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:56:09.863577 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0005d91_7470_4ec7_9087_a53710e3e94c.slice/crio-e3b3957228fb4c2d79006ceb8c575847fe225706c3aa18195b0cb9d39873c388 WatchSource:0}: Error finding container e3b3957228fb4c2d79006ceb8c575847fe225706c3aa18195b0cb9d39873c388: Status 404 returned error can't find the container with id e3b3957228fb4c2d79006ceb8c575847fe225706c3aa18195b0cb9d39873c388 Nov 02 19:56:09.865505 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.865438 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn"] Nov 02 19:56:09.891797 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:09.891768 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:10.029497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:10.029406 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:10.191029 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:10.190996 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:10.240359 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:10.240320 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" event={"ID":"b0005d91-7470-4ec7-9087-a53710e3e94c","Type":"ContainerStarted","Data":"0c2499eff28e65ecaecb38e30f7dbd2f7d40c92c41e911f63b0b11ca7ce34ec8"} Nov 02 19:56:10.240586 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:10.240381 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" event={"ID":"b0005d91-7470-4ec7-9087-a53710e3e94c","Type":"ContainerStarted","Data":"e3b3957228fb4c2d79006ceb8c575847fe225706c3aa18195b0cb9d39873c388"} Nov 02 19:56:10.240586 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:10.240545 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:10.363618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:10.363555 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" podStartSLOduration=2.363534139 podStartE2EDuration="2.363534139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:10.36335984 +0000 UTC m=+1628.705748920" watchObservedRunningTime="2026-11-02 19:56:10.363534139 +0000 UTC m=+1628.705923224" Nov 02 19:56:10.379187 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:56:10.379150 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e71fad_4636_4a1f_a2e4_a5bd787d7eba.slice/crio-3ef10960a5673dc689c7f9b72ba3eaeece50b19bf1133b1e22b626a21180e19d WatchSource:0}: Error finding container 3ef10960a5673dc689c7f9b72ba3eaeece50b19bf1133b1e22b626a21180e19d: Status 404 returned error can't find the container with id 3ef10960a5673dc689c7f9b72ba3eaeece50b19bf1133b1e22b626a21180e19d Nov 02 19:56:10.384959 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:10.384932 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp"] Nov 02 19:56:10.582401 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:10.582364 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:11.249101 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:11.249060 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" event={"ID":"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba","Type":"ContainerStarted","Data":"bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18"} Nov 02 19:56:11.249534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:11.249127 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" event={"ID":"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba","Type":"ContainerStarted","Data":"3ef10960a5673dc689c7f9b72ba3eaeece50b19bf1133b1e22b626a21180e19d"} Nov 02 19:56:11.249534 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:11.249370 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:11.367077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:11.367022 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" podStartSLOduration=3.367007873 podStartE2EDuration="3.367007873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:11.364917495 +0000 UTC m=+1629.707306588" watchObservedRunningTime="2026-11-02 19:56:11.367007873 +0000 UTC m=+1629.709396952" Nov 02 19:56:12.532643 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:12.526526 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:13.074663 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:13.074627 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:13.674642 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:13.674605 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:13.879498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:13.875794 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:14.769506 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:56:14.769431 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:56:16.547812 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:16.547777 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:16.712624 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:16.712583 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:16.884727 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:16.884693 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:17.672083 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:17.672051 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:17.829717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:17.829617 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:18.284570 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.284533 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m"] Nov 02 19:56:18.289276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.289250 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.348500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.346202 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b0cc24-openshift-default-dockercfg-hd8fw\"" Nov 02 19:56:18.392496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.385890 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-socket\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.392496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.385945 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-credential-socket\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.392496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.386002 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05973ae4-c92f-460b-a550-76b094ea96fd-istio-podinfo\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.392496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.386051 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxq9\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-kube-api-access-8cxq9\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.392496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.386079 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-data\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.392496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.386116 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-certs\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.392496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.386230 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05973ae4-c92f-460b-a550-76b094ea96fd-istiod-ca-cert\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.392496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.386261 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-envoy\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.392496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.386354 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-istio-token\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.440498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.430924 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.500783 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-socket\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.500835 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-credential-socket\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.500890 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05973ae4-c92f-460b-a550-76b094ea96fd-istio-podinfo\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.500934 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxq9\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-kube-api-access-8cxq9\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.500965 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-data\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.500996 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-certs\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.501108 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05973ae4-c92f-460b-a550-76b094ea96fd-istiod-ca-cert\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.501137 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-envoy\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.501219 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-istio-token\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.502128 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-socket\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.502500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.502462 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-credential-socket\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.518833 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.503901 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-certs\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.518833 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.504419 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-data\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.518833 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.505099 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05973ae4-c92f-460b-a550-76b094ea96fd-istiod-ca-cert\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.568513 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.568327 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-istio-token\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.581892 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.580906 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxq9\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-kube-api-access-8cxq9\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.618648 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.618525 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05973ae4-c92f-460b-a550-76b094ea96fd-istio-podinfo\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.631087 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.630683 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-envoy\") pod \"e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:18.726070 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.716967 5850 generic.go:356] "Generic (PLEG): container finished" podID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerID="7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54" exitCode=0 Nov 02 19:56:18.726070 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.717110 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" event={"ID":"71cb4481-97cd-44e3-a14a-01dbcac2752c","Type":"ContainerDied","Data":"7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54"} Nov 02 19:56:18.726070 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.717671 5850 scope.go:122] "RemoveContainer" containerID="7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54" Nov 02 19:56:18.726070 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.719648 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:18.726070 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.719758 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:18.726070 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.719814 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:18.768741 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.735162 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m"] Nov 02 19:56:18.792551 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.792413 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:18.904653 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:18.904613 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:19.353075 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:56:19.353028 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05973ae4_c92f_460b_a550_76b094ea96fd.slice/crio-3735fe521f1b8cbaab9093af947a6464963ac44ba5a97cc092e57dcc515aef35 WatchSource:0}: Error finding container 3735fe521f1b8cbaab9093af947a6464963ac44ba5a97cc092e57dcc515aef35: Status 404 returned error can't find the container with id 3735fe521f1b8cbaab9093af947a6464963ac44ba5a97cc092e57dcc515aef35 Nov 02 19:56:19.356106 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:19.356062 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:19.356216 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:19.356140 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:19.356216 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:19.356174 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:19.356621 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:19.356595 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m"] Nov 02 19:56:19.725122 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:19.725081 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" event={"ID":"05973ae4-c92f-460b-a550-76b094ea96fd","Type":"ContainerStarted","Data":"e0d9484c96f5f1a4530576e5c01b2e102173787f8d64fe2d924fdf731b80bb71"} Nov 02 19:56:19.725310 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:19.725148 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" event={"ID":"05973ae4-c92f-460b-a550-76b094ea96fd","Type":"ContainerStarted","Data":"3735fe521f1b8cbaab9093af947a6464963ac44ba5a97cc092e57dcc515aef35"} Nov 02 19:56:19.727323 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:19.727297 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" event={"ID":"71cb4481-97cd-44e3-a14a-01dbcac2752c","Type":"ContainerStarted","Data":"556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6"} Nov 02 19:56:19.844185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:19.844132 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" podStartSLOduration=1.844118683 podStartE2EDuration="1.844118683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:19.843154464 +0000 UTC m=+1638.185543545" watchObservedRunningTime="2026-11-02 19:56:19.844118683 +0000 UTC m=+1638.186507762" Nov 02 19:56:19.905199 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:19.905155 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:20.364111 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:20.364075 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:56:20.365010 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:20.364987 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 19:56:20.365084 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:20.365038 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 19:56:20.925498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:20.916882 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:21.252611 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:21.252525 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 19:56:21.363642 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:21.363589 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 19:56:21.363816 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:21.363657 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 19:56:21.745452 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:21.745424 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:21.746636 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:21.746614 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:56:22.260535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:22.260502 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 19:56:22.281411 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:22.281373 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:22.446499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:22.445751 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 19:56:22.446499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:22.445809 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 19:56:22.561424 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:22.561342 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:56:22.648622 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:22.640909 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:22.830760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:22.830671 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:23.365187 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:23.365147 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 19:56:23.365635 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:23.365213 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 19:56:24.364120 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:24.364082 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 19:56:24.364284 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:24.364149 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 19:56:25.364374 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:25.364337 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 19:56:25.364864 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:25.364409 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 19:56:25.896280 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:25.896240 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:26.117730 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.116924 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:26.370042 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.370011 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:56:26.371324 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.371302 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 19:56:26.799859 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.799771 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb"] Nov 02 19:56:26.804261 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.804237 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.857987 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.857941 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-32abda-serving-cert\"" Nov 02 19:56:26.863925 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.863891 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb"] Nov 02 19:56:26.894315 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.894271 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bjdw\" (UniqueName: \"kubernetes.io/projected/4408de12-184d-4185-9f4e-b5eda75b116a-kube-api-access-2bjdw\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.894559 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.894352 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4408de12-184d-4185-9f4e-b5eda75b116a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.894724 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.894686 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.894939 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.894837 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.996709 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.996677 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.996877 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.996739 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.996877 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.996839 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bjdw\" (UniqueName: \"kubernetes.io/projected/4408de12-184d-4185-9f4e-b5eda75b116a-kube-api-access-2bjdw\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.997131 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.996878 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4408de12-184d-4185-9f4e-b5eda75b116a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.997131 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:56:26.996883 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-serving-cert: secret "maas-api-e2e-shared-b-w5-32abda-serving-cert" not found Nov 02 19:56:26.997131 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:56:26.996949 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls podName:4408de12-184d-4185-9f4e-b5eda75b116a nodeName:}" failed. No retries permitted until 2026-11-02 19:56:27.49693098 +0000 UTC m=+1645.839320041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls") pod "maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" (UID: "4408de12-184d-4185-9f4e-b5eda75b116a") : secret "maas-api-e2e-shared-b-w5-32abda-serving-cert" not found Nov 02 19:56:26.997580 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.997561 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4408de12-184d-4185-9f4e-b5eda75b116a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:26.999121 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:26.999103 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:27.050226 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:27.050139 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:27.056248 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:27.056218 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bjdw\" (UniqueName: \"kubernetes.io/projected/4408de12-184d-4185-9f4e-b5eda75b116a-kube-api-access-2bjdw\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:27.211138 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:27.211026 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:27.374729 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:27.374697 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:27.505003 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:27.504968 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:27.507672 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:27.507638 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:27.730178 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:27.730086 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:28.108549 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:56:28.108506 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4408de12_184d_4185_9f4e_b5eda75b116a.slice/crio-2318698ff3f090868562a054daa8a6468c9354c85216c70c943caa402307b84e WatchSource:0}: Error finding container 2318698ff3f090868562a054daa8a6468c9354c85216c70c943caa402307b84e: Status 404 returned error can't find the container with id 2318698ff3f090868562a054daa8a6468c9354c85216c70c943caa402307b84e Nov 02 19:56:28.110296 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:28.110269 5850 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:56:28.111910 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:28.111787 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb"] Nov 02 19:56:28.799860 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:28.799817 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" event={"ID":"4408de12-184d-4185-9f4e-b5eda75b116a","Type":"ContainerStarted","Data":"2318698ff3f090868562a054daa8a6468c9354c85216c70c943caa402307b84e"} Nov 02 19:56:29.924499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:29.924092 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" event={"ID":"4408de12-184d-4185-9f4e-b5eda75b116a","Type":"ContainerStarted","Data":"62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b"} Nov 02 19:56:29.924499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:29.924151 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:30.032868 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:30.032786 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" podStartSLOduration=2.889182656 podStartE2EDuration="4.032766632s" totalImagesPullingTime="1.143583976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:30.030676575 +0000 UTC m=+1648.373065656" watchObservedRunningTime="2026-11-02 19:56:30.032766632 +0000 UTC m=+1648.375155712" Nov 02 19:56:31.487326 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:31.487281 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:31.985297 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:31.985236 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:35.965321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:35.965289 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 19:56:36.520276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:36.516239 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:36.717906 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:36.717868 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:37.622010 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:37.621979 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:40.939376 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:40.939337 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:42.128031 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:42.127991 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:42.651623 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:42.651563 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:42.804831 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:42.804798 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:43.226702 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:43.226668 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:43.643664 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:43.643620 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:44.763187 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:56:44.763151 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:56:45.745959 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:45.745920 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:47.893308 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:47.893275 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:48.054748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:48.054693 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:48.221567 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:48.221457 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:50.465651 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:50.465612 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:51.192366 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:51.192337 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:51.385622 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:51.385567 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:55.326642 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:56:55.326609 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:00.012657 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:00.012608 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:02.616523 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:02.587522 5850 scope.go:122] "RemoveContainer" containerID="2d38ab58692f35950a3042b6c571193ef3adf6308fea6ad60b6df3fe16675329" Nov 02 19:57:02.943500 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:02.941862 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:57:03.097503 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:03.094300 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq" podUID="db3b64b1-d8e5-4e64-a5c4-97158401ed5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:57:03.121580 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:03.118810 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 19:57:03.121580 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:03.119856 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:57:03.142501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:03.141134 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:57:03.193498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:03.193072 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:57:03.197498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:03.193873 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:57:03.201604 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:03.201553 5850 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:57:03.455916 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:03.455722 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:57:04.572789 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:04.572754 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:08.698717 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.698673 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj"] Nov 02 19:57:08.705873 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.705843 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.757872 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.757836 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj"] Nov 02 19:57:08.805047 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.805009 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0b7eed-serving-cert\"" Nov 02 19:57:08.807915 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.807881 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c204339b-7ab7-43b1-afff-456ac421b761-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.808064 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.807995 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-metrics-certs\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.808142 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.808090 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7v9n\" (UniqueName: \"kubernetes.io/projected/c204339b-7ab7-43b1-afff-456ac421b761-kube-api-access-r7v9n\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.808275 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.808259 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.899191 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.899156 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:08.909677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.909640 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c204339b-7ab7-43b1-afff-456ac421b761-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.909859 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.909699 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-metrics-certs\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.909859 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.909764 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7v9n\" (UniqueName: \"kubernetes.io/projected/c204339b-7ab7-43b1-afff-456ac421b761-kube-api-access-r7v9n\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.910027 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.909920 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.910085 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:57:08.910024 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-serving-cert: secret "maas-api-e2e-api-a-0b7eed-serving-cert" not found Nov 02 19:57:08.910137 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:57:08.910092 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls podName:c204339b-7ab7-43b1-afff-456ac421b761 nodeName:}" failed. No retries permitted until 2026-11-02 19:57:09.410071804 +0000 UTC m=+1687.752460871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls") pod "maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" (UID: "c204339b-7ab7-43b1-afff-456ac421b761") : secret "maas-api-e2e-api-a-0b7eed-serving-cert" not found Nov 02 19:57:08.910819 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.910795 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c204339b-7ab7-43b1-afff-456ac421b761-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.912820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.912789 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-metrics-certs\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:08.991424 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:08.991204 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7v9n\" (UniqueName: \"kubernetes.io/projected/c204339b-7ab7-43b1-afff-456ac421b761-kube-api-access-r7v9n\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:09.056403 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:09.056320 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:09.260796 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:09.260718 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:09.422276 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:09.422224 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:09.424738 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:09.424712 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls\") pod \"maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:09.621234 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:09.621195 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:10.004492 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:57:10.004446 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc204339b_7ab7_43b1_afff_456ac421b761.slice/crio-687ab93c82be0eff4957bfc0135c2072429ece7f5852740e56390fd127be3879 WatchSource:0}: Error finding container 687ab93c82be0eff4957bfc0135c2072429ece7f5852740e56390fd127be3879: Status 404 returned error can't find the container with id 687ab93c82be0eff4957bfc0135c2072429ece7f5852740e56390fd127be3879 Nov 02 19:57:10.007585 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:10.007395 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj"] Nov 02 19:57:10.409865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:10.409833 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" event={"ID":"c204339b-7ab7-43b1-afff-456ac421b761","Type":"ContainerStarted","Data":"687ab93c82be0eff4957bfc0135c2072429ece7f5852740e56390fd127be3879"} Nov 02 19:57:11.418420 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:11.418377 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" event={"ID":"c204339b-7ab7-43b1-afff-456ac421b761","Type":"ContainerStarted","Data":"c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13"} Nov 02 19:57:11.418940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:11.418526 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:11.420136 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:11.420109 5850 generic.go:356] "Generic (PLEG): container finished" podID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerID="6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630" exitCode=0 Nov 02 19:57:11.420264 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:11.420178 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" event={"ID":"cec5c949-3bd9-4ff6-a7be-76c6bf064239","Type":"ContainerDied","Data":"6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630"} Nov 02 19:57:11.420593 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:11.420579 5850 scope.go:122] "RemoveContainer" containerID="6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630" Nov 02 19:57:11.422185 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:11.422159 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:57:11.422250 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:11.422223 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:57:11.422285 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:11.422258 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:57:11.573497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:11.568784 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" podStartSLOduration=2.590430381 podStartE2EDuration="3.568759773s" totalImagesPullingTime="978.329392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:11.566624916 +0000 UTC m=+1689.909013996" watchObservedRunningTime="2026-11-02 19:57:11.568759773 +0000 UTC m=+1689.911148932" Nov 02 19:57:12.345964 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:12.345925 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:57:12.440563 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:12.440527 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" event={"ID":"cec5c949-3bd9-4ff6-a7be-76c6bf064239","Type":"ContainerStarted","Data":"5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7"} Nov 02 19:57:12.631708 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:12.631663 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:57:12.632932 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:12.632909 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 19:57:12.633017 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:12.632957 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 19:57:12.951618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:12.951494 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:13.391413 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:13.391376 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:13.548496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:13.547598 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:13.631498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:13.630769 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 19:57:13.631498 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:13.630927 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 19:57:13.785386 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:13.785315 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:14.631497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:14.630949 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 19:57:14.631497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:14.631023 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 19:57:14.763448 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:57:14.763409 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:57:15.632911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:15.632869 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 19:57:15.633539 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:15.633495 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 19:57:16.630956 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:16.630912 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 19:57:16.631279 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:16.631249 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 19:57:17.447767 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:17.447734 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 19:57:17.637179 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:17.637137 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 19:57:17.637345 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:17.637203 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 19:57:18.036711 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:18.036676 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:18.291715 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:18.291631 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:18.635558 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:18.635524 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:57:18.636005 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:18.635992 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:57:18.636677 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:18.636660 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 19:57:22.466153 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:22.466123 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:26.847087 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:26.847053 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:31.117214 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:31.117174 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:31.289366 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:31.285718 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:35.727497 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:35.726575 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:37.306309 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:37.306267 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25"] Nov 02 19:57:37.306865 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:37.306618 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" containerID="cri-o://3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67" gracePeriod=30 Nov 02 19:57:38.060872 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:38.060837 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:57:38.061052 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:38.060896 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:57:38.782998 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:38.782956 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:40.256940 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:40.256845 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:42.640348 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.640317 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:57:42.642661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.642637 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-kube-api-access-q899p\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-kube-api-access-q899p\") pod \"0867b967-5ce9-4bca-908f-b9a9bffaa601\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " Nov 02 19:57:42.642805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.642700 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0867b967-5ce9-4bca-908f-b9a9bffaa601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867b967-5ce9-4bca-908f-b9a9bffaa601-istiod-ca-cert\") pod \"0867b967-5ce9-4bca-908f-b9a9bffaa601\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " Nov 02 19:57:42.642805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.642757 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-credential-socket\") pod \"0867b967-5ce9-4bca-908f-b9a9bffaa601\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " Nov 02 19:57:42.642970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.642802 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-envoy\") pod \"0867b967-5ce9-4bca-908f-b9a9bffaa601\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " Nov 02 19:57:42.642970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.642836 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-socket\") pod \"0867b967-5ce9-4bca-908f-b9a9bffaa601\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " Nov 02 19:57:42.642970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.642871 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-token\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-token\") pod \"0867b967-5ce9-4bca-908f-b9a9bffaa601\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " Nov 02 19:57:42.642970 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.642954 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-data\") pod \"0867b967-5ce9-4bca-908f-b9a9bffaa601\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " Nov 02 19:57:42.644164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.643205 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-socket" pod "0867b967-5ce9-4bca-908f-b9a9bffaa601" (UID: "0867b967-5ce9-4bca-908f-b9a9bffaa601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.644164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.643556 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-data" pod "0867b967-5ce9-4bca-908f-b9a9bffaa601" (UID: "0867b967-5ce9-4bca-908f-b9a9bffaa601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.644164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.643747 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-credential-socket" pod "0867b967-5ce9-4bca-908f-b9a9bffaa601" (UID: "0867b967-5ce9-4bca-908f-b9a9bffaa601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.644164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.643784 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-podinfo\") pod \"0867b967-5ce9-4bca-908f-b9a9bffaa601\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " Nov 02 19:57:42.644164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.644158 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-certs\") pod \"0867b967-5ce9-4bca-908f-b9a9bffaa601\" (UID: \"0867b967-5ce9-4bca-908f-b9a9bffaa601\") " Nov 02 19:57:42.644845 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.644822 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0867b967-5ce9-4bca-908f-b9a9bffaa601-istiod-ca-cert" pod "0867b967-5ce9-4bca-908f-b9a9bffaa601" (UID: "0867b967-5ce9-4bca-908f-b9a9bffaa601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:57:42.644989 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.644932 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.645215 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.645168 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.645370 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.645356 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.646192 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.646026 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-certs" pod "0867b967-5ce9-4bca-908f-b9a9bffaa601" (UID: "0867b967-5ce9-4bca-908f-b9a9bffaa601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.647086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.647043 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-envoy" pod "0867b967-5ce9-4bca-908f-b9a9bffaa601" (UID: "0867b967-5ce9-4bca-908f-b9a9bffaa601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.647180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.647131 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-podinfo" pod "0867b967-5ce9-4bca-908f-b9a9bffaa601" (UID: "0867b967-5ce9-4bca-908f-b9a9bffaa601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:57:42.647180 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.647134 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-kube-api-access-q899p" pod "0867b967-5ce9-4bca-908f-b9a9bffaa601" (UID: "0867b967-5ce9-4bca-908f-b9a9bffaa601"). InnerVolumeSpecName "kube-api-access-q899p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:42.647457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.647427 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-token" pod "0867b967-5ce9-4bca-908f-b9a9bffaa601" (UID: "0867b967-5ce9-4bca-908f-b9a9bffaa601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:42.747518 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.747469 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.747518 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.747515 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.747518 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.747526 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q899p\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-kube-api-access-q899p\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.747750 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.747537 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867b967-5ce9-4bca-908f-b9a9bffaa601-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.747750 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.747546 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.747750 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.747554 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0867b967-5ce9-4bca-908f-b9a9bffaa601-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.779633 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.779591 5850 generic.go:356] "Generic (PLEG): container finished" podID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerID="3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67" exitCode=0 Nov 02 19:57:42.779840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.779734 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" event={"ID":"0867b967-5ce9-4bca-908f-b9a9bffaa601","Type":"ContainerDied","Data":"3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67"} Nov 02 19:57:42.779840 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.779823 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" event={"ID":"0867b967-5ce9-4bca-908f-b9a9bffaa601","Type":"ContainerDied","Data":"f39ef07075a0387f73137c6e8aefeff4a2366406fb3df74dc9cb2bc60c4c968e"} Nov 02 19:57:42.779973 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.779853 5850 scope.go:122] "RemoveContainer" containerID="3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67" Nov 02 19:57:42.780533 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.780022 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25" Nov 02 19:57:42.796785 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.796756 5850 scope.go:122] "RemoveContainer" containerID="5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c" Nov 02 19:57:42.811780 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.811695 5850 scope.go:122] "RemoveContainer" containerID="3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67" Nov 02 19:57:42.812293 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:57:42.812262 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67\": container with ID starting with 3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67 not found: ID does not exist" containerID="3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67" Nov 02 19:57:42.813514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.812738 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67"} err="failed to get container status \"3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67\": rpc error: code = NotFound desc = could not find container \"3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67\": container with ID starting with 3f93dc7baa5b368281a44e45ef8b418d06fa3354d1f128b8b28f7becdd3e5a67 not found: ID does not exist" Nov 02 19:57:42.813514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.812780 5850 scope.go:122] "RemoveContainer" containerID="5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c" Nov 02 19:57:42.813708 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:57:42.813605 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c\": container with ID starting with 5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c not found: ID does not exist" containerID="5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c" Nov 02 19:57:42.813708 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.813639 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c"} err="failed to get container status \"5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c\": rpc error: code = NotFound desc = could not find container \"5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c\": container with ID starting with 5a5e597e06efe101262ee169c82a112616e3c1be7aa6bc35cf83b10d1bd3e58c not found: ID does not exist" Nov 02 19:57:42.957134 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.957097 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25"] Nov 02 19:57:42.973779 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:42.973745 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-gmw25"] Nov 02 19:57:43.660254 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:43.660219 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:43.820844 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:43.820798 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" path="/var/lib/kubelet/pods/0867b967-5ce9-4bca-908f-b9a9bffaa601/volumes" Nov 02 19:57:44.707065 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:44.706659 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:44.808981 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:57:44.808946 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:57:45.943349 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:45.943286 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:46.814373 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:46.814281 5850 generic.go:356] "Generic (PLEG): container finished" podID="69514c97-be51-45d0-818f-29c4d2e69318" containerID="4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d" exitCode=0 Nov 02 19:57:46.814373 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:46.814354 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" event={"ID":"69514c97-be51-45d0-818f-29c4d2e69318","Type":"ContainerDied","Data":"4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d"} Nov 02 19:57:46.814825 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:46.814810 5850 scope.go:122] "RemoveContainer" containerID="4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d" Nov 02 19:57:46.816522 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:46.816493 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:57:46.816592 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:46.816564 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:57:46.816636 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:46.816596 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:57:47.136659 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:47.136618 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:47.784671 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:47.784625 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:57:47.890184 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:47.890119 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" event={"ID":"69514c97-be51-45d0-818f-29c4d2e69318","Type":"ContainerStarted","Data":"21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650"} Nov 02 19:57:48.433702 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:48.433662 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:48.634151 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:48.634096 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:48.695174 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:48.689178 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:57:48.722431 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:48.722394 5850 patch_prober.go:29] interesting pod/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 19:57:48.740624 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:48.740578 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 19:57:49.430854 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:49.430815 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:49.675974 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:49.675927 5850 patch_prober.go:29] interesting pod/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 19:57:49.676490 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:49.675993 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 19:57:50.675351 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:50.675313 5850 patch_prober.go:29] interesting pod/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 19:57:50.675571 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:50.675383 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 19:57:51.675937 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:51.675898 5850 patch_prober.go:29] interesting pod/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 19:57:51.676334 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:51.675956 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 19:57:52.599054 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:52.599015 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:52.675900 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:52.675849 5850 patch_prober.go:29] interesting pod/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 19:57:52.676082 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:52.675926 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 19:57:52.901936 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:52.901883 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:53.679932 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:53.679901 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:57:53.680360 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:53.680343 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:57:53.681015 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:53.680998 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 19:57:54.170932 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:54.170896 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:55.867195 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:55.867149 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:56.044956 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:56.044921 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:58.926397 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:57:58.926350 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:03.231938 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:03.231840 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:03.394447 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:03.394409 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:07.825113 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:07.825075 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:10.291135 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:10.291080 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:11.950799 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:11.950758 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:12.304655 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:12.304567 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:14.431541 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:14.412181 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:14.762617 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:58:14.762516 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:58:16.744468 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:16.744432 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:16.932525 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:16.932435 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:17.156536 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:17.150053 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:18.036560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:18.036519 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:20.018905 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:20.018871 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:20.209988 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:20.209951 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:21.623530 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:21.623495 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:22.788644 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:22.788601 5850 patch_prober.go:29] interesting pod/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:58:22.789224 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:22.789186 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:58:23.707791 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:23.707752 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:25.359148 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:25.359114 5850 generic.go:356] "Generic (PLEG): container finished" podID="05973ae4-c92f-460b-a550-76b094ea96fd" containerID="e0d9484c96f5f1a4530576e5c01b2e102173787f8d64fe2d924fdf731b80bb71" exitCode=0 Nov 02 19:58:25.359728 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:25.359189 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" event={"ID":"05973ae4-c92f-460b-a550-76b094ea96fd","Type":"ContainerDied","Data":"e0d9484c96f5f1a4530576e5c01b2e102173787f8d64fe2d924fdf731b80bb71"} Nov 02 19:58:25.359798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:25.359786 5850 scope.go:122] "RemoveContainer" containerID="e0d9484c96f5f1a4530576e5c01b2e102173787f8d64fe2d924fdf731b80bb71" Nov 02 19:58:25.361697 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:25.361661 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:25.361837 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:25.361733 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:25.361837 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:25.361763 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:26.052301 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:26.052263 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:26.368347 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:26.368309 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" event={"ID":"05973ae4-c92f-460b-a550-76b094ea96fd","Type":"ContainerStarted","Data":"fef0fc9919cc9fe80619b5db859024d3517c592f304993f9ceae8b46ba6e755a"} Nov 02 19:58:26.905549 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:26.905509 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:58:26.905772 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:26.905750 5850 patch_prober.go:29] interesting pod/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 19:58:26.905828 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:26.905810 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 19:58:27.927968 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:27.927924 5850 patch_prober.go:29] interesting pod/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 19:58:27.931661 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:27.931616 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 19:58:28.905593 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:28.905555 5850 patch_prober.go:29] interesting pod/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 19:58:28.905812 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:28.905611 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 19:58:29.910123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:29.910084 5850 patch_prober.go:29] interesting pod/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 19:58:29.912542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:29.912503 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 19:58:30.535395 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:30.535302 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:30.905398 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:30.905367 5850 patch_prober.go:29] interesting pod/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 19:58:30.905639 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:30.905423 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 19:58:31.905433 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:31.905389 5850 patch_prober.go:29] interesting pod/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 19:58:31.905895 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:31.905525 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 19:58:32.632780 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:32.632693 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:32.910393 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:32.910306 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:58:32.910981 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:32.910961 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:58:32.911917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:32.911893 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 19:58:33.432514 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:33.432482 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:34.109501 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:34.076458 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:34.303859 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:34.303726 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:35.590502 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:35.583965 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:38.670588 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:38.669624 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:38.869499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:38.868926 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:39.057560 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:39.052063 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:39.229154 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:39.229118 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:40.456739 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:40.456687 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:58:40.499849 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:40.499809 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:40.532493 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:40.523853 5850 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:58:40.936496 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:40.936251 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 02 19:58:40.962780 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:40.954559 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:41.126571 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:41.126513 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:58:41.128527 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:41.128417 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:58:41.178053 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:41.178009 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 19:58:41.548908 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:41.548826 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:42.427834 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:42.427797 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:42.617618 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:42.617582 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:44.763236 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:58:44.763197 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:58:45.675762 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:45.675729 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:46.214760 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:46.214725 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:46.394606 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:46.394568 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:46.557805 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:46.557702 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:47.135287 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:47.135256 5850 generic.go:356] "Generic (PLEG): container finished" podID="aa77d63a-7ab5-4e82-9144-15039098659b" containerID="d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc" exitCode=0 Nov 02 19:58:47.135469 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:47.135325 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" event={"ID":"aa77d63a-7ab5-4e82-9144-15039098659b","Type":"ContainerDied","Data":"d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc"} Nov 02 19:58:47.135817 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:47.135803 5850 scope.go:122] "RemoveContainer" containerID="d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc" Nov 02 19:58:47.137747 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:47.137715 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:47.137848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:47.137789 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:47.137848 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:47.137825 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:58:48.147379 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:48.147347 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" event={"ID":"aa77d63a-7ab5-4e82-9144-15039098659b","Type":"ContainerStarted","Data":"d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb"} Nov 02 19:58:49.099547 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:49.099506 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:58:49.100719 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:49.100694 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 19:58:49.100845 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:49.100749 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 19:58:49.935099 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:49.935058 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:58:50.099691 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:50.099652 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 19:58:50.099905 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:50.099709 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 19:58:50.197752 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:50.197665 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:50.429660 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:50.414194 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:50.754499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:50.740994 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:51.129535 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:51.124255 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 19:58:51.153167 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:51.153126 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 19:58:52.101746 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:52.101711 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 19:58:52.102081 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:52.102035 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 19:58:53.099586 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:53.099546 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 19:58:53.099989 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:53.099614 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 19:58:54.104391 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:54.104361 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:58:54.105433 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:54.105411 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 19:58:55.415892 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:58:55.415857 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:00.080231 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:00.080197 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:00.849239 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:00.849204 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:01.239647 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:01.239609 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:02.067310 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:02.067268 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:02.220154 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:02.220118 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:02.367077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:02.367039 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:04.869521 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:04.869485 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:06.363930 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:06.363892 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:06.529090 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:06.529049 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:06.700377 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:06.700345 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:09.631818 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:09.631736 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:09.824711 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:09.820160 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:13.600346 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:13.600311 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:14.183843 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.183803 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl"] Nov 02 19:59:14.184626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.184604 5850 cpu_manager.go:414] "containerMap: removing container" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" Nov 02 19:59:14.184626 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.184624 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" Nov 02 19:59:14.184863 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.184850 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" Nov 02 19:59:14.184922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.184876 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" Nov 02 19:59:14.187761 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.187742 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.244733 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.244696 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl"] Nov 02 19:59:14.291942 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.291907 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8479ff-serving-cert\"" Nov 02 19:59:14.316887 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.316848 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.317066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.316898 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-metrics-certs\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.317066 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.317038 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.317177 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.317105 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvtd\" (UniqueName: \"kubernetes.io/projected/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-kube-api-access-ksvtd\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.394298 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.394256 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:14.427345 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.427287 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.432064 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.432027 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-metrics-certs\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.433077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.433053 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.433454 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.433435 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvtd\" (UniqueName: \"kubernetes.io/projected/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-kube-api-access-ksvtd\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.433765 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:59:14.433305 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-serving-cert: secret "maas-api-e2e-inf-a-8479ff-serving-cert" not found Nov 02 19:59:14.433966 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:59:14.433919 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls podName:75ef5acc-2c2f-4d74-9aa2-721859e4ac1f nodeName:}" failed. No retries permitted until 2026-11-02 19:59:14.933893091 +0000 UTC m=+1813.276282166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls") pod "maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" (UID: "75ef5acc-2c2f-4d74-9aa2-721859e4ac1f") : secret "maas-api-e2e-inf-a-8479ff-serving-cert" not found Nov 02 19:59:14.434328 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.428203 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.438040 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.438001 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-metrics-certs\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.497820 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.497761 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvtd\" (UniqueName: \"kubernetes.io/projected/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-kube-api-access-ksvtd\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.549695 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.549655 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:14.696229 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.696144 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:14.763362 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:59:14.763320 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:59:14.943546 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.943507 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:14.946340 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:14.946268 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls\") pod \"maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:15.102987 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:15.102954 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:15.508415 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:59:15.508374 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ef5acc_2c2f_4d74_9aa2_721859e4ac1f.slice/crio-5f7f3c5850f54522f4831fc9b5635df6b65b4f1285ae53b4f403e673b961645d WatchSource:0}: Error finding container 5f7f3c5850f54522f4831fc9b5635df6b65b4f1285ae53b4f403e673b961645d: Status 404 returned error can't find the container with id 5f7f3c5850f54522f4831fc9b5635df6b65b4f1285ae53b4f403e673b961645d Nov 02 19:59:15.511387 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:15.511358 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl"] Nov 02 19:59:16.510457 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:16.510409 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" event={"ID":"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f","Type":"ContainerStarted","Data":"5f7f3c5850f54522f4831fc9b5635df6b65b4f1285ae53b4f403e673b961645d"} Nov 02 19:59:17.518886 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:17.518849 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" event={"ID":"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f","Type":"ContainerStarted","Data":"60bcecb5ba5fd464c65c0422d60680e442eba16ddd7f7ca11b7f3161824eb4bf"} Nov 02 19:59:17.519364 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:17.518993 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:17.671922 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:17.671865 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" podStartSLOduration=2.703361088 podStartE2EDuration="3.671850496s" totalImagesPullingTime="968.489408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:17.668535552 +0000 UTC m=+1816.010924631" watchObservedRunningTime="2026-11-02 19:59:17.671850496 +0000 UTC m=+1816.014239577" Nov 02 19:59:19.250118 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:19.250085 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:23.532164 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:23.532088 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 19:59:23.955854 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:23.955810 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:28.345441 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:28.345400 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:32.759761 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:32.759723 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:36.438020 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:36.437981 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:37.440939 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:37.440889 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:42.181732 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:42.181691 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:42.341373 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:42.341335 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:42.504903 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:42.504825 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:44.762305 ip-10-0-131-244 kubenswrapper[5850]: E1102 19:59:44.762263 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 19:59:47.014410 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:47.014376 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:47.883128 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:47.883096 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:48.387638 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.387601 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27"] Nov 02 19:59:48.388449 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.388423 5850 cpu_manager.go:414] "containerMap: removing container" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" Nov 02 19:59:48.388449 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.388446 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0867b967-5ce9-4bca-908f-b9a9bffaa601" containerName="istio-proxy" Nov 02 19:59:48.391542 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.391523 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.487499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.480125 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:48.487499 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.485760 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7ab11604-openshift-default-dockercfg-t7lsd\"" Nov 02 19:59:48.661079 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.660980 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istiod-ca-cert\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.802089 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-token\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.802156 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-socket\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.802310 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-podinfo\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.802354 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-data\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.802493 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg95h\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-kube-api-access-hg95h\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.802557 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-credential-socket\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.802616 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-certs\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.802679 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-envoy\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.730966 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:48.827058 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.779739 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27"] Nov 02 19:59:48.919411 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.918575 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istiod-ca-cert\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.920282 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.919333 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istiod-ca-cert\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.920846 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.920341 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-token\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.921077 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.921052 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-socket\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.921602 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.921533 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-socket\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.921911 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.921895 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-podinfo\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.922467 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.922449 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-data\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.923107 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.922843 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-data\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.923188 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.923093 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg95h\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-kube-api-access-hg95h\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.923245 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.923218 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-credential-socket\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.923311 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.923294 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-certs\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.923423 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.923364 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-envoy\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.923782 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.923756 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-certs\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.924023 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.924002 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-credential-socket\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.940383 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.926561 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-podinfo\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.940383 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.926976 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-envoy\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.967730 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.967687 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:48.978384 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.978356 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-token\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:48.983898 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:48.983872 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg95h\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-kube-api-access-hg95h\") pod \"e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:49.038748 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.038714 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:49.564985 ip-10-0-131-244 kubenswrapper[5850]: W1102 19:59:49.564949 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a00dd58_66dc_4d13_97b8_acfcf52513e9.slice/crio-813271b8648310d72dcd3f0f7348ff25a97f98713ae1bdbdd24b282841455f7d WatchSource:0}: Error finding container 813271b8648310d72dcd3f0f7348ff25a97f98713ae1bdbdd24b282841455f7d: Status 404 returned error can't find the container with id 813271b8648310d72dcd3f0f7348ff25a97f98713ae1bdbdd24b282841455f7d Nov 02 19:59:49.568136 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.568096 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:49.568224 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.568167 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:49.568224 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.568202 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:59:49.568543 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.568521 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27"] Nov 02 19:59:49.602392 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.602354 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:49.783321 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.783281 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" event={"ID":"1a00dd58-66dc-4d13-97b8-acfcf52513e9","Type":"ContainerStarted","Data":"6d84bc8142d363c4ae6aaf1879076088ce79afa56a8ca07241ed90782672fe69"} Nov 02 19:59:49.783527 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.783340 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" event={"ID":"1a00dd58-66dc-4d13-97b8-acfcf52513e9","Type":"ContainerStarted","Data":"813271b8648310d72dcd3f0f7348ff25a97f98713ae1bdbdd24b282841455f7d"} Nov 02 19:59:49.786757 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.786720 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:49.909086 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:49.909020 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" podStartSLOduration=1.9090022119999999 podStartE2EDuration="1.909002212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:49.905630432 +0000 UTC m=+1848.248019532" watchObservedRunningTime="2026-11-02 19:59:49.909002212 +0000 UTC m=+1848.251391294" Nov 02 19:59:50.039425 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:50.039389 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:50.040744 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:50.040717 5850 patch_prober.go:29] interesting pod/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 19:59:50.040836 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:50.040779 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 19:59:51.044123 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:51.044083 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:51.541016 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:51.540971 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:51.797513 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:51.797397 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:51.798384 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:51.798366 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 19:59:52.698798 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:52.698759 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:52.862803 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:52.862772 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:53.062914 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:53.062821 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:54.015917 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:54.015881 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:56.010812 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:56.010781 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:56.195605 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:56.195420 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:56.349600 ip-10-0-131-244 kubenswrapper[5850]: I1102 19:59:56.349502 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:00.212605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.212571 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr"] Nov 02 20:00:00.216321 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.216294 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k"] Nov 02 20:00:00.216453 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.216410 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:00.219892 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.219869 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:00.234356 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.234327 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d"] Nov 02 20:00:00.237836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.237817 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:00.324561 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.324521 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr"] Nov 02 20:00:00.327703 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.327675 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:00:00.332765 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.332729 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2c7j\" (UniqueName: \"kubernetes.io/projected/c54d0bae-85b4-437d-bdae-9d2486ec98e1-kube-api-access-h2c7j\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d\" (UID: \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:00.332923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.332799 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr\" (UID: \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:00.332923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.332873 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8gb\" (UniqueName: \"kubernetes.io/projected/1887850b-06fb-4dc3-b3e2-5ee558704129-kube-api-access-2q8gb\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k\" (UID: \"1887850b-06fb-4dc3-b3e2-5ee558704129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:00.333074 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.332977 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1887850b-06fb-4dc3-b3e2-5ee558704129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k\" (UID: \"1887850b-06fb-4dc3-b3e2-5ee558704129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:00.333317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.333292 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54d0bae-85b4-437d-bdae-9d2486ec98e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d\" (UID: \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:00.333412 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.333391 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2pxx\" (UniqueName: \"kubernetes.io/projected/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-kube-api-access-c2pxx\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr\" (UID: \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:00.384693 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.384650 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k"] Nov 02 20:00:00.435359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.435322 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1887850b-06fb-4dc3-b3e2-5ee558704129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k\" (UID: \"1887850b-06fb-4dc3-b3e2-5ee558704129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:00.435618 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.435598 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54d0bae-85b4-437d-bdae-9d2486ec98e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d\" (UID: \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:00.435726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.435663 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2pxx\" (UniqueName: \"kubernetes.io/projected/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-kube-api-access-c2pxx\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr\" (UID: \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:00.435726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.435717 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2c7j\" (UniqueName: \"kubernetes.io/projected/c54d0bae-85b4-437d-bdae-9d2486ec98e1-kube-api-access-h2c7j\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d\" (UID: \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:00.435850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.435763 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr\" (UID: \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:00.435850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.435799 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8gb\" (UniqueName: \"kubernetes.io/projected/1887850b-06fb-4dc3-b3e2-5ee558704129-kube-api-access-2q8gb\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k\" (UID: \"1887850b-06fb-4dc3-b3e2-5ee558704129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:00.436236 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.436205 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1887850b-06fb-4dc3-b3e2-5ee558704129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k\" (UID: \"1887850b-06fb-4dc3-b3e2-5ee558704129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:00.436731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.436708 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr\" (UID: \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:00.436828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.436807 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54d0bae-85b4-437d-bdae-9d2486ec98e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d\" (UID: \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:00.443928 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.443899 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d"] Nov 02 20:00:00.504120 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.504044 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2c7j\" (UniqueName: \"kubernetes.io/projected/c54d0bae-85b4-437d-bdae-9d2486ec98e1-kube-api-access-h2c7j\") pod \"maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d\" (UID: \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:00.504120 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.504059 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2pxx\" (UniqueName: \"kubernetes.io/projected/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-kube-api-access-c2pxx\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr\" (UID: \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:00.504940 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.504913 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8gb\" (UniqueName: \"kubernetes.io/projected/1887850b-06fb-4dc3-b3e2-5ee558704129-kube-api-access-2q8gb\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k\" (UID: \"1887850b-06fb-4dc3-b3e2-5ee558704129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:00.536433 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.536397 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:00.543491 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.543453 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:00.554006 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.553967 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:00.882909 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.882859 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:00.994900 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:00:00.993507 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952f9f92_6ba2_4aac_93f1_a55cd17f98b0.slice/crio-55ce507e9cedabd89a5283a1ae3273d5f693c81f212895c58ace2bf1d9f25909 WatchSource:0}: Error finding container 55ce507e9cedabd89a5283a1ae3273d5f693c81f212895c58ace2bf1d9f25909: Status 404 returned error can't find the container with id 55ce507e9cedabd89a5283a1ae3273d5f693c81f212895c58ace2bf1d9f25909 Nov 02 20:00:00.999968 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:00.999929 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr"] Nov 02 20:00:01.008027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.008001 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k"] Nov 02 20:00:01.012746 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.012719 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d"] Nov 02 20:00:01.585835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.585807 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:01.750605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.750568 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:01.871631 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.871593 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" event={"ID":"1887850b-06fb-4dc3-b3e2-5ee558704129","Type":"ContainerStarted","Data":"4fb9346962fd556f8a016e73c3a2eb7f08f145272019ce5c270d958fb9f043b4"} Nov 02 20:00:01.871824 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.871656 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" event={"ID":"1887850b-06fb-4dc3-b3e2-5ee558704129","Type":"ContainerStarted","Data":"ae29a67acff6523bd6e968667ecc4b44205bca6cffa550e41986be84f0837295"} Nov 02 20:00:01.873248 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.873221 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" event={"ID":"c54d0bae-85b4-437d-bdae-9d2486ec98e1","Type":"ContainerStarted","Data":"d37d6e9fb4579373dccad17931549c4c9f90260d3b8b10eb9f7a819f602c3f49"} Nov 02 20:00:01.873393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.873273 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" event={"ID":"c54d0bae-85b4-437d-bdae-9d2486ec98e1","Type":"ContainerStarted","Data":"6c3889c48b357eb1c1b7b5d2714268645a612d4fdbce011a38348c257e016c98"} Nov 02 20:00:01.874984 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.874959 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" event={"ID":"952f9f92-6ba2-4aac-93f1-a55cd17f98b0","Type":"ContainerStarted","Data":"cd50e1c0b65dbc4d658e6e785a27b446ddf10774f7c01a9c2c8424d0390ec737"} Nov 02 20:00:01.875099 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.875006 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" event={"ID":"952f9f92-6ba2-4aac-93f1-a55cd17f98b0","Type":"ContainerStarted","Data":"55ce507e9cedabd89a5283a1ae3273d5f693c81f212895c58ace2bf1d9f25909"} Nov 02 20:00:01.900844 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.900803 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:01.991075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:01.991013 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" podStartSLOduration=1.990992748 podStartE2EDuration="1.990992748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:01.990632969 +0000 UTC m=+1860.333022048" watchObservedRunningTime="2026-11-02 20:00:01.990992748 +0000 UTC m=+1860.333381829" Nov 02 20:00:02.112686 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.112626 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" podStartSLOduration=2.112607841 podStartE2EDuration="2.112607841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:02.107030271 +0000 UTC m=+1860.449419352" watchObservedRunningTime="2026-11-02 20:00:02.112607841 +0000 UTC m=+1860.454996983" Nov 02 20:00:02.885129 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.885094 5850 generic.go:356] "Generic (PLEG): container finished" podID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerID="4fb9346962fd556f8a016e73c3a2eb7f08f145272019ce5c270d958fb9f043b4" exitCode=60 Nov 02 20:00:02.885615 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.885173 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" event={"ID":"1887850b-06fb-4dc3-b3e2-5ee558704129","Type":"ContainerDied","Data":"4fb9346962fd556f8a016e73c3a2eb7f08f145272019ce5c270d958fb9f043b4"} Nov 02 20:00:02.885615 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.885455 5850 scope.go:122] "RemoveContainer" containerID="4fb9346962fd556f8a016e73c3a2eb7f08f145272019ce5c270d958fb9f043b4" Nov 02 20:00:02.887030 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.887008 5850 generic.go:356] "Generic (PLEG): container finished" podID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerID="d37d6e9fb4579373dccad17931549c4c9f90260d3b8b10eb9f7a819f602c3f49" exitCode=60 Nov 02 20:00:02.887150 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.887086 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" event={"ID":"c54d0bae-85b4-437d-bdae-9d2486ec98e1","Type":"ContainerDied","Data":"d37d6e9fb4579373dccad17931549c4c9f90260d3b8b10eb9f7a819f602c3f49"} Nov 02 20:00:02.887306 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.887290 5850 scope.go:122] "RemoveContainer" containerID="d37d6e9fb4579373dccad17931549c4c9f90260d3b8b10eb9f7a819f602c3f49" Nov 02 20:00:02.888879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.888860 5850 generic.go:356] "Generic (PLEG): container finished" podID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerID="cd50e1c0b65dbc4d658e6e785a27b446ddf10774f7c01a9c2c8424d0390ec737" exitCode=60 Nov 02 20:00:02.889010 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.888942 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" event={"ID":"952f9f92-6ba2-4aac-93f1-a55cd17f98b0","Type":"ContainerDied","Data":"cd50e1c0b65dbc4d658e6e785a27b446ddf10774f7c01a9c2c8424d0390ec737"} Nov 02 20:00:02.889232 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:02.889211 5850 scope.go:122] "RemoveContainer" containerID="cd50e1c0b65dbc4d658e6e785a27b446ddf10774f7c01a9c2c8424d0390ec737" Nov 02 20:00:03.004563 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:03.004494 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" podStartSLOduration=3.00446424 podStartE2EDuration="3.00446424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:02.228656683 +0000 UTC m=+1860.571045775" watchObservedRunningTime="2026-11-02 20:00:03.00446424 +0000 UTC m=+1861.346853397" Nov 02 20:00:03.898463 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:03.898425 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" event={"ID":"1887850b-06fb-4dc3-b3e2-5ee558704129","Type":"ContainerStarted","Data":"14617ab7a6cfe40a89964f89539ac5ed60dd68a6f09d137a5e4dd835d3b7a272"} Nov 02 20:00:03.900413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:03.900383 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" event={"ID":"c54d0bae-85b4-437d-bdae-9d2486ec98e1","Type":"ContainerStarted","Data":"4ba8329a3db5ce3cad79e95084caf38e17a49124779e928aac101243eb4c9c71"} Nov 02 20:00:03.902508 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:03.902466 5850 generic.go:356] "Generic (PLEG): container finished" podID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerID="319fb0b14ab6aebff45fda3289f0518a294c9a368cb75a85ee9b5f1500aad325" exitCode=60 Nov 02 20:00:03.902643 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:03.902503 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" event={"ID":"952f9f92-6ba2-4aac-93f1-a55cd17f98b0","Type":"ContainerDied","Data":"319fb0b14ab6aebff45fda3289f0518a294c9a368cb75a85ee9b5f1500aad325"} Nov 02 20:00:03.902643 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:03.902573 5850 scope.go:122] "RemoveContainer" containerID="cd50e1c0b65dbc4d658e6e785a27b446ddf10774f7c01a9c2c8424d0390ec737" Nov 02 20:00:03.902820 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:03.902793 5850 scope.go:122] "RemoveContainer" containerID="319fb0b14ab6aebff45fda3289f0518a294c9a368cb75a85ee9b5f1500aad325" Nov 02 20:00:03.903156 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:03.903127 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr_odh-ai-gateway-infra(952f9f92-6ba2-4aac-93f1-a55cd17f98b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" Nov 02 20:00:04.874369 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.874328 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:04.913259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.913215 5850 generic.go:356] "Generic (PLEG): container finished" podID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerID="14617ab7a6cfe40a89964f89539ac5ed60dd68a6f09d137a5e4dd835d3b7a272" exitCode=60 Nov 02 20:00:04.913813 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.913298 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" event={"ID":"1887850b-06fb-4dc3-b3e2-5ee558704129","Type":"ContainerDied","Data":"14617ab7a6cfe40a89964f89539ac5ed60dd68a6f09d137a5e4dd835d3b7a272"} Nov 02 20:00:04.913813 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.913370 5850 scope.go:122] "RemoveContainer" containerID="4fb9346962fd556f8a016e73c3a2eb7f08f145272019ce5c270d958fb9f043b4" Nov 02 20:00:04.913813 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.913664 5850 scope.go:122] "RemoveContainer" containerID="14617ab7a6cfe40a89964f89539ac5ed60dd68a6f09d137a5e4dd835d3b7a272" Nov 02 20:00:04.914131 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:04.914103 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k_odh-ai-gateway-infra(1887850b-06fb-4dc3-b3e2-5ee558704129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" Nov 02 20:00:04.915703 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.915679 5850 generic.go:356] "Generic (PLEG): container finished" podID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerID="4ba8329a3db5ce3cad79e95084caf38e17a49124779e928aac101243eb4c9c71" exitCode=60 Nov 02 20:00:04.915796 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.915761 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" event={"ID":"c54d0bae-85b4-437d-bdae-9d2486ec98e1","Type":"ContainerDied","Data":"4ba8329a3db5ce3cad79e95084caf38e17a49124779e928aac101243eb4c9c71"} Nov 02 20:00:04.916147 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.916125 5850 scope.go:122] "RemoveContainer" containerID="4ba8329a3db5ce3cad79e95084caf38e17a49124779e928aac101243eb4c9c71" Nov 02 20:00:04.916611 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:04.916585 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d_odh-ai-gateway-infra(c54d0bae-85b4-437d-bdae-9d2486ec98e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" Nov 02 20:00:04.918585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.918562 5850 scope.go:122] "RemoveContainer" containerID="319fb0b14ab6aebff45fda3289f0518a294c9a368cb75a85ee9b5f1500aad325" Nov 02 20:00:04.919052 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:04.918842 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr_odh-ai-gateway-infra(952f9f92-6ba2-4aac-93f1-a55cd17f98b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" Nov 02 20:00:04.929401 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:04.929368 5850 scope.go:122] "RemoveContainer" containerID="d37d6e9fb4579373dccad17931549c4c9f90260d3b8b10eb9f7a819f602c3f49" Nov 02 20:00:05.057587 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:05.057551 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:05.928079 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:05.928045 5850 scope.go:122] "RemoveContainer" containerID="14617ab7a6cfe40a89964f89539ac5ed60dd68a6f09d137a5e4dd835d3b7a272" Nov 02 20:00:05.928563 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:05.928506 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k_odh-ai-gateway-infra(1887850b-06fb-4dc3-b3e2-5ee558704129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" Nov 02 20:00:05.929735 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:05.929718 5850 scope.go:122] "RemoveContainer" containerID="4ba8329a3db5ce3cad79e95084caf38e17a49124779e928aac101243eb4c9c71" Nov 02 20:00:05.930033 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:05.930015 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d_odh-ai-gateway-infra(c54d0bae-85b4-437d-bdae-9d2486ec98e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" Nov 02 20:00:06.033457 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:06.033421 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:08.227020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:08.226980 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6"] Nov 02 20:00:08.227449 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:08.227200 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" containerID="cri-o://408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f" gracePeriod=30 Nov 02 20:00:08.278873 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:08.278828 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st"] Nov 02 20:00:08.279155 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:08.279128 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" containerID="cri-o://14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2" gracePeriod=30 Nov 02 20:00:08.284029 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:08.283990 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979"] Nov 02 20:00:08.284301 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:08.284273 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" containerID="cri-o://79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631" gracePeriod=30 Nov 02 20:00:08.521912 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:08.521813 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb"] Nov 02 20:00:08.522149 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:08.522119 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" containerID="cri-o://0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1" gracePeriod=30 Nov 02 20:00:10.305541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.305495 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb"] Nov 02 20:00:10.310837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.310816 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.376189 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.376151 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb"] Nov 02 20:00:10.386262 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.386224 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59l87\" (UniqueName: \"kubernetes.io/projected/2e11a43c-4a44-4d8f-ab4d-099257c7443a-kube-api-access-59l87\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.386626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.386591 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.386847 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.386825 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e11a43c-4a44-4d8f-ab4d-099257c7443a-openshift-service-ca\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.387040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.387015 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-metrics-certs\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.416633 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.416591 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c60a0d-serving-cert\"" Nov 02 20:00:10.493640 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.493597 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59l87\" (UniqueName: \"kubernetes.io/projected/2e11a43c-4a44-4d8f-ab4d-099257c7443a-kube-api-access-59l87\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.493829 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.493679 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.493875 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:10.493832 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-serving-cert: secret "maas-api-e2e-api-b-c60a0d-serving-cert" not found Nov 02 20:00:10.493995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.493882 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e11a43c-4a44-4d8f-ab4d-099257c7443a-openshift-service-ca\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.493995 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:10.493909 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls podName:2e11a43c-4a44-4d8f-ab4d-099257c7443a nodeName:}" failed. No retries permitted until 2026-11-02 20:00:10.993891071 +0000 UTC m=+1869.336280147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls") pod "maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" (UID: "2e11a43c-4a44-4d8f-ab4d-099257c7443a") : secret "maas-api-e2e-api-b-c60a0d-serving-cert" not found Nov 02 20:00:10.493995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.493967 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-metrics-certs\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.494533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.494511 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e11a43c-4a44-4d8f-ab4d-099257c7443a-openshift-service-ca\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.496764 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.496737 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-metrics-certs\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.509661 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.509634 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:10.566807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.566715 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59l87\" (UniqueName: \"kubernetes.io/projected/2e11a43c-4a44-4d8f-ab4d-099257c7443a-kube-api-access-59l87\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:10.653612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.653576 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:10.841892 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:10.841803 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:11.010805 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:11.010766 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:11.013251 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:11.013227 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls\") pod \"maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:11.112057 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:11.111976 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:00:11.112057 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:11.112030 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:00:11.224942 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:11.224906 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:11.604982 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:00:11.604947 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e11a43c_4a44_4d8f_ab4d_099257c7443a.slice/crio-90af404e348f14e5cc42f1a5a0f50c8a8cf9a0a1d76457066168dd57fc0f8548 WatchSource:0}: Error finding container 90af404e348f14e5cc42f1a5a0f50c8a8cf9a0a1d76457066168dd57fc0f8548: Status 404 returned error can't find the container with id 90af404e348f14e5cc42f1a5a0f50c8a8cf9a0a1d76457066168dd57fc0f8548 Nov 02 20:00:11.606529 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:11.606502 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb"] Nov 02 20:00:11.975924 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:11.975886 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" event={"ID":"2e11a43c-4a44-4d8f-ab4d-099257c7443a","Type":"ContainerStarted","Data":"90af404e348f14e5cc42f1a5a0f50c8a8cf9a0a1d76457066168dd57fc0f8548"} Nov 02 20:00:12.984769 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:12.984731 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" event={"ID":"2e11a43c-4a44-4d8f-ab4d-099257c7443a","Type":"ContainerStarted","Data":"7d69d7bdcd923290a4c10246501b3e772fa3d0f986448b866db50120b40feafd"} Nov 02 20:00:12.985195 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:12.984872 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:13.110503 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.108258 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" podStartSLOduration=2.208414587 podStartE2EDuration="3.10823744s" totalImagesPullingTime="899.822853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:13.103586893 +0000 UTC m=+1871.445975976" watchObservedRunningTime="2026-11-02 20:00:13.10823744 +0000 UTC m=+1871.450626520" Nov 02 20:00:13.647738 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.647708 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 20:00:13.652425 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.652397 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 20:00:13.665765 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.665729 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 20:00:13.781643 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781614 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44370a98-8413-404c-b45e-bf9bbc523424-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44370a98-8413-404c-b45e-bf9bbc523424-istio-podinfo\") pod \"44370a98-8413-404c-b45e-bf9bbc523424\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " Nov 02 20:00:13.781935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781665 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-credential-socket\") pod \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " Nov 02 20:00:13.781935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781687 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-token\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-token\") pod \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " Nov 02 20:00:13.781935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781763 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-socket\") pod \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " Nov 02 20:00:13.781935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781784 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-token\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-token\") pod \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " Nov 02 20:00:13.781935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781865 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-socket\") pod \"44370a98-8413-404c-b45e-bf9bbc523424\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " Nov 02 20:00:13.781935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781899 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istiod-ca-cert\") pod \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " Nov 02 20:00:13.781935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781934 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-kube-api-access-4vxxg\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-kube-api-access-4vxxg\") pod \"44370a98-8413-404c-b45e-bf9bbc523424\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781939 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-credential-socket" pod "58af3e1a-198c-4706-9bfa-e5b1ba476d4c" (UID: "58af3e1a-198c-4706-9bfa-e5b1ba476d4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.781961 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-credential-socket\") pod \"44370a98-8413-404c-b45e-bf9bbc523424\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782000 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-certs\") pod \"44370a98-8413-404c-b45e-bf9bbc523424\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782037 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-socket\") pod \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782065 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-podinfo\") pod \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782087 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-kube-api-access-l2k8x\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-kube-api-access-l2k8x\") pod \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782123 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-envoy\") pod \"44370a98-8413-404c-b45e-bf9bbc523424\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782149 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-envoy\") pod \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782195 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-data\") pod \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782254 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-credential-socket\") pod \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782284 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44370a98-8413-404c-b45e-bf9bbc523424-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44370a98-8413-404c-b45e-bf9bbc523424-istiod-ca-cert\") pod \"44370a98-8413-404c-b45e-bf9bbc523424\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782298 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-socket" pod "44370a98-8413-404c-b45e-bf9bbc523424" (UID: "44370a98-8413-404c-b45e-bf9bbc523424"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782307 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-socket" pod "268a0dec-6781-41b8-b2c9-fe08f8d18ded" (UID: "268a0dec-6781-41b8-b2c9-fe08f8d18ded"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782319 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istiod-ca-cert\") pod \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782394 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-kube-api-access-8wdqv\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-kube-api-access-8wdqv\") pod \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782418 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-data\") pod \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " Nov 02 20:00:13.782455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782448 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-istio-token\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-istio-token\") pod \"44370a98-8413-404c-b45e-bf9bbc523424\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " Nov 02 20:00:13.783383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782549 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-envoy\") pod \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " Nov 02 20:00:13.783383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782571 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-data\") pod \"44370a98-8413-404c-b45e-bf9bbc523424\" (UID: \"44370a98-8413-404c-b45e-bf9bbc523424\") " Nov 02 20:00:13.783383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782597 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-podinfo\") pod \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " Nov 02 20:00:13.783383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782620 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istiod-ca-cert" pod "268a0dec-6781-41b8-b2c9-fe08f8d18ded" (UID: "268a0dec-6781-41b8-b2c9-fe08f8d18ded"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:13.783383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782643 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-certs\") pod \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\" (UID: \"58af3e1a-198c-4706-9bfa-e5b1ba476d4c\") " Nov 02 20:00:13.783383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.782715 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-certs\") pod \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\" (UID: \"268a0dec-6781-41b8-b2c9-fe08f8d18ded\") " Nov 02 20:00:13.783383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783006 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-credential-socket" pod "44370a98-8413-404c-b45e-bf9bbc523424" (UID: "44370a98-8413-404c-b45e-bf9bbc523424"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.783383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783316 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istiod-ca-cert" pod "58af3e1a-198c-4706-9bfa-e5b1ba476d4c" (UID: "58af3e1a-198c-4706-9bfa-e5b1ba476d4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:13.783853 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783545 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-certs" pod "268a0dec-6781-41b8-b2c9-fe08f8d18ded" (UID: "268a0dec-6781-41b8-b2c9-fe08f8d18ded"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.783932 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783855 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.783932 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783877 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.783932 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783894 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.784075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783934 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.784075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783949 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.784075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783962 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.784075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.783979 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.785033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.784299 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44370a98-8413-404c-b45e-bf9bbc523424-istio-podinfo" pod "44370a98-8413-404c-b45e-bf9bbc523424" (UID: "44370a98-8413-404c-b45e-bf9bbc523424"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:00:13.785033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.784575 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-data" pod "58af3e1a-198c-4706-9bfa-e5b1ba476d4c" (UID: "58af3e1a-198c-4706-9bfa-e5b1ba476d4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.785033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.784615 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-credential-socket" pod "268a0dec-6781-41b8-b2c9-fe08f8d18ded" (UID: "268a0dec-6781-41b8-b2c9-fe08f8d18ded"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.785258 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.785119 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-kube-api-access-l2k8x" pod "58af3e1a-198c-4706-9bfa-e5b1ba476d4c" (UID: "58af3e1a-198c-4706-9bfa-e5b1ba476d4c"). InnerVolumeSpecName "kube-api-access-l2k8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:13.785519 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.785458 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-token" pod "58af3e1a-198c-4706-9bfa-e5b1ba476d4c" (UID: "58af3e1a-198c-4706-9bfa-e5b1ba476d4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:13.785594 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.785550 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44370a98-8413-404c-b45e-bf9bbc523424-istiod-ca-cert" pod "44370a98-8413-404c-b45e-bf9bbc523424" (UID: "44370a98-8413-404c-b45e-bf9bbc523424"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:13.785930 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.785902 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-token" pod "268a0dec-6781-41b8-b2c9-fe08f8d18ded" (UID: "268a0dec-6781-41b8-b2c9-fe08f8d18ded"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:13.786828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.786206 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-certs" pod "58af3e1a-198c-4706-9bfa-e5b1ba476d4c" (UID: "58af3e1a-198c-4706-9bfa-e5b1ba476d4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.786828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.786293 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-podinfo" pod "58af3e1a-198c-4706-9bfa-e5b1ba476d4c" (UID: "58af3e1a-198c-4706-9bfa-e5b1ba476d4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:00:13.786828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.786464 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-data" pod "44370a98-8413-404c-b45e-bf9bbc523424" (UID: "44370a98-8413-404c-b45e-bf9bbc523424"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.786828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.786586 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-certs" pod "44370a98-8413-404c-b45e-bf9bbc523424" (UID: "44370a98-8413-404c-b45e-bf9bbc523424"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.786828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.786630 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-kube-api-access-4vxxg" pod "44370a98-8413-404c-b45e-bf9bbc523424" (UID: "44370a98-8413-404c-b45e-bf9bbc523424"). InnerVolumeSpecName "kube-api-access-4vxxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:13.786828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.786705 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-data" pod "268a0dec-6781-41b8-b2c9-fe08f8d18ded" (UID: "268a0dec-6781-41b8-b2c9-fe08f8d18ded"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.787212 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.786865 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-socket" pod "58af3e1a-198c-4706-9bfa-e5b1ba476d4c" (UID: "58af3e1a-198c-4706-9bfa-e5b1ba476d4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.787212 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.786882 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-envoy" pod "44370a98-8413-404c-b45e-bf9bbc523424" (UID: "44370a98-8413-404c-b45e-bf9bbc523424"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.787212 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.787112 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-envoy" pod "58af3e1a-198c-4706-9bfa-e5b1ba476d4c" (UID: "58af3e1a-198c-4706-9bfa-e5b1ba476d4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.787350 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.787249 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-envoy" pod "268a0dec-6781-41b8-b2c9-fe08f8d18ded" (UID: "268a0dec-6781-41b8-b2c9-fe08f8d18ded"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.787767 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.787745 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-podinfo" pod "268a0dec-6781-41b8-b2c9-fe08f8d18ded" (UID: "268a0dec-6781-41b8-b2c9-fe08f8d18ded"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:00:13.788355 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.788334 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 20:00:13.788589 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.788569 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-kube-api-access-8wdqv" pod "268a0dec-6781-41b8-b2c9-fe08f8d18ded" (UID: "268a0dec-6781-41b8-b2c9-fe08f8d18ded"). InnerVolumeSpecName "kube-api-access-8wdqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:13.788729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.788711 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-istio-token" pod "44370a98-8413-404c-b45e-bf9bbc523424" (UID: "44370a98-8413-404c-b45e-bf9bbc523424"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:13.882354 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.882315 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:13.887301 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887263 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-kube-api-access-l82m9\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-kube-api-access-l82m9\") pod \"64524fe5-c084-40bd-9195-7ca010e99c99\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " Nov 02 20:00:13.887555 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887364 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64524fe5-c084-40bd-9195-7ca010e99c99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64524fe5-c084-40bd-9195-7ca010e99c99-istiod-ca-cert\") pod \"64524fe5-c084-40bd-9195-7ca010e99c99\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " Nov 02 20:00:13.887555 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887400 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-data\") pod \"64524fe5-c084-40bd-9195-7ca010e99c99\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " Nov 02 20:00:13.887555 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887429 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-certs\") pod \"64524fe5-c084-40bd-9195-7ca010e99c99\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " Nov 02 20:00:13.887734 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887563 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64524fe5-c084-40bd-9195-7ca010e99c99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64524fe5-c084-40bd-9195-7ca010e99c99-istio-podinfo\") pod \"64524fe5-c084-40bd-9195-7ca010e99c99\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " Nov 02 20:00:13.887734 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887600 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-socket\") pod \"64524fe5-c084-40bd-9195-7ca010e99c99\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " Nov 02 20:00:13.887734 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887722 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-istio-token\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-istio-token\") pod \"64524fe5-c084-40bd-9195-7ca010e99c99\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " Nov 02 20:00:13.887872 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887844 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-envoy\") pod \"64524fe5-c084-40bd-9195-7ca010e99c99\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " Nov 02 20:00:13.887923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887877 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-credential-socket\") pod \"64524fe5-c084-40bd-9195-7ca010e99c99\" (UID: \"64524fe5-c084-40bd-9195-7ca010e99c99\") " Nov 02 20:00:13.887984 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887934 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-certs" pod "64524fe5-c084-40bd-9195-7ca010e99c99" (UID: "64524fe5-c084-40bd-9195-7ca010e99c99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.888047 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.887999 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-data" pod "64524fe5-c084-40bd-9195-7ca010e99c99" (UID: "64524fe5-c084-40bd-9195-7ca010e99c99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.888148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.888103 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-socket" pod "64524fe5-c084-40bd-9195-7ca010e99c99" (UID: "64524fe5-c084-40bd-9195-7ca010e99c99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.888320 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.888295 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64524fe5-c084-40bd-9195-7ca010e99c99-istiod-ca-cert" pod "64524fe5-c084-40bd-9195-7ca010e99c99" (UID: "64524fe5-c084-40bd-9195-7ca010e99c99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:13.888419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.888395 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-credential-socket" pod "64524fe5-c084-40bd-9195-7ca010e99c99" (UID: "64524fe5-c084-40bd-9195-7ca010e99c99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.891843 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.891814 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-envoy" pod "64524fe5-c084-40bd-9195-7ca010e99c99" (UID: "64524fe5-c084-40bd-9195-7ca010e99c99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:13.891988 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.891882 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64524fe5-c084-40bd-9195-7ca010e99c99-istio-podinfo" pod "64524fe5-c084-40bd-9195-7ca010e99c99" (UID: "64524fe5-c084-40bd-9195-7ca010e99c99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:00:13.892074 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.891941 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-kube-api-access-l82m9" pod "64524fe5-c084-40bd-9195-7ca010e99c99" (UID: "64524fe5-c084-40bd-9195-7ca010e99c99"). InnerVolumeSpecName "kube-api-access-l82m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:13.892138 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892065 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-istio-token" pod "64524fe5-c084-40bd-9195-7ca010e99c99" (UID: "64524fe5-c084-40bd-9195-7ca010e99c99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:13.892468 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892445 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64524fe5-c084-40bd-9195-7ca010e99c99-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892487 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892503 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44370a98-8413-404c-b45e-bf9bbc523424-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892517 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892531 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892544 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892559 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vxxg\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-kube-api-access-4vxxg\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892574 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892587 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892601 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892615 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892628 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2k8x\" (UniqueName: \"kubernetes.io/projected/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-kube-api-access-l2k8x\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892642 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892655 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892668 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892680 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892694 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892708 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44370a98-8413-404c-b45e-bf9bbc523424-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892722 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wdqv\" (UniqueName: \"kubernetes.io/projected/268a0dec-6781-41b8-b2c9-fe08f8d18ded-kube-api-access-8wdqv\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892735 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892750 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44370a98-8413-404c-b45e-bf9bbc523424-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892764 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l82m9\" (UniqueName: \"kubernetes.io/projected/64524fe5-c084-40bd-9195-7ca010e99c99-kube-api-access-l82m9\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892778 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892793 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44370a98-8413-404c-b45e-bf9bbc523424-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892808 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a0dec-6781-41b8-b2c9-fe08f8d18ded-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892822 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58af3e1a-198c-4706-9bfa-e5b1ba476d4c-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892835 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64524fe5-c084-40bd-9195-7ca010e99c99-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892850 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.892880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.892864 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64524fe5-c084-40bd-9195-7ca010e99c99-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:13.994289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.994251 5850 generic.go:356] "Generic (PLEG): container finished" podID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerID="408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f" exitCode=0 Nov 02 20:00:13.994761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.994308 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" Nov 02 20:00:13.994761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.994327 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" event={"ID":"268a0dec-6781-41b8-b2c9-fe08f8d18ded","Type":"ContainerDied","Data":"408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f"} Nov 02 20:00:13.994761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.994405 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6" event={"ID":"268a0dec-6781-41b8-b2c9-fe08f8d18ded","Type":"ContainerDied","Data":"63469727a28e4d89710bd66c787ae63dec62ab8750652cb0b8363e0575ca2f98"} Nov 02 20:00:13.994761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.994432 5850 scope.go:122] "RemoveContainer" containerID="408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f" Nov 02 20:00:13.996456 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.996414 5850 generic.go:356] "Generic (PLEG): container finished" podID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerID="14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2" exitCode=0 Nov 02 20:00:13.996630 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.996533 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" Nov 02 20:00:13.996847 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.996702 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" event={"ID":"58af3e1a-198c-4706-9bfa-e5b1ba476d4c","Type":"ContainerDied","Data":"14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2"} Nov 02 20:00:13.996847 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.996756 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" event={"ID":"58af3e1a-198c-4706-9bfa-e5b1ba476d4c","Type":"ContainerDied","Data":"59b02093da115b8bd1f152b888d52def3a15b795b9f729c04d6ca99230522351"} Nov 02 20:00:13.998393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.998366 5850 generic.go:356] "Generic (PLEG): container finished" podID="64524fe5-c084-40bd-9195-7ca010e99c99" containerID="0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1" exitCode=0 Nov 02 20:00:13.998579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.998559 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" event={"ID":"64524fe5-c084-40bd-9195-7ca010e99c99","Type":"ContainerDied","Data":"0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1"} Nov 02 20:00:13.998647 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.998605 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" event={"ID":"64524fe5-c084-40bd-9195-7ca010e99c99","Type":"ContainerDied","Data":"4ac030e3aba92f36e66d211033ef02d3f0f238ddaf947facb5c10e2bb756b795"} Nov 02 20:00:13.998702 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:13.998683 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb" Nov 02 20:00:14.003529 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.003455 5850 generic.go:356] "Generic (PLEG): container finished" podID="44370a98-8413-404c-b45e-bf9bbc523424" containerID="79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631" exitCode=0 Nov 02 20:00:14.004561 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.004539 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" Nov 02 20:00:14.005174 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.005150 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" event={"ID":"44370a98-8413-404c-b45e-bf9bbc523424","Type":"ContainerDied","Data":"79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631"} Nov 02 20:00:14.005913 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.005884 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979" event={"ID":"44370a98-8413-404c-b45e-bf9bbc523424","Type":"ContainerDied","Data":"19dba6f9f1bf12dc2cad54d72242c9e229bd65944e9a3d9bd5d0c5f75c2efdda"} Nov 02 20:00:14.008990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.008962 5850 scope.go:122] "RemoveContainer" containerID="408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f" Nov 02 20:00:14.009514 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:14.009483 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f\": container with ID starting with 408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f not found: ID does not exist" containerID="408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f" Nov 02 20:00:14.009611 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.009526 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f"} err="failed to get container status \"408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f\": rpc error: code = NotFound desc = could not find container \"408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f\": container with ID starting with 408eac78171abe23fb6376693157634bc94dfc8fb09c8f12e8cb085542ef286f not found: ID does not exist" Nov 02 20:00:14.009611 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.009550 5850 scope.go:122] "RemoveContainer" containerID="14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2" Nov 02 20:00:14.021464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.021436 5850 scope.go:122] "RemoveContainer" containerID="14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2" Nov 02 20:00:14.021805 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:14.021782 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2\": container with ID starting with 14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2 not found: ID does not exist" containerID="14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2" Nov 02 20:00:14.021901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.021816 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2"} err="failed to get container status \"14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2\": rpc error: code = NotFound desc = could not find container \"14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2\": container with ID starting with 14c11241fcba03020906c27a008782eb4f9f9297c577853aa9ed050b85453dc2 not found: ID does not exist" Nov 02 20:00:14.021901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.021842 5850 scope.go:122] "RemoveContainer" containerID="0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1" Nov 02 20:00:14.032017 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.031996 5850 scope.go:122] "RemoveContainer" containerID="0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1" Nov 02 20:00:14.032317 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:14.032291 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1\": container with ID starting with 0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1 not found: ID does not exist" containerID="0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1" Nov 02 20:00:14.032385 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.032326 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1"} err="failed to get container status \"0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1\": rpc error: code = NotFound desc = could not find container \"0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1\": container with ID starting with 0b6d760c6145d8979a57357705d8136d98c0227a218fa1f90790692475bd3ff1 not found: ID does not exist" Nov 02 20:00:14.032385 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.032346 5850 scope.go:122] "RemoveContainer" containerID="79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631" Nov 02 20:00:14.042423 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.042400 5850 scope.go:122] "RemoveContainer" containerID="79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631" Nov 02 20:00:14.042766 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:14.042744 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631\": container with ID starting with 79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631 not found: ID does not exist" containerID="79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631" Nov 02 20:00:14.042831 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.042777 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631"} err="failed to get container status \"79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631\": rpc error: code = NotFound desc = could not find container \"79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631\": container with ID starting with 79b4552a1581b3c004f8746f27d253a79af44366ed632392a017fc54695c1631 not found: ID does not exist" Nov 02 20:00:14.306563 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.306525 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979"] Nov 02 20:00:14.322383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.312707 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-rl979"] Nov 02 20:00:14.424773 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.424735 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb"] Nov 02 20:00:14.428760 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.428728 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-pmrlb"] Nov 02 20:00:14.435906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.435880 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:14.542188 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.542134 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st"] Nov 02 20:00:14.545362 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.545340 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st"] Nov 02 20:00:14.579063 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.578968 5850 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:00:14.579063 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.579048 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hz7st" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:00:14.598285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.598251 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:14.657347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.657303 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6"] Nov 02 20:00:14.660919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.660885 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-sgbj6"] Nov 02 20:00:14.761081 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.761046 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:14.763968 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:14.763927 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:00:14.923771 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:14.923739 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:15.155270 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:15.155235 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:15.808757 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:15.808713 5850 scope.go:122] "RemoveContainer" containerID="4ba8329a3db5ce3cad79e95084caf38e17a49124779e928aac101243eb4c9c71" Nov 02 20:00:15.821240 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:15.821205 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" path="/var/lib/kubelet/pods/268a0dec-6781-41b8-b2c9-fe08f8d18ded/volumes" Nov 02 20:00:15.821864 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:15.821844 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44370a98-8413-404c-b45e-bf9bbc523424" path="/var/lib/kubelet/pods/44370a98-8413-404c-b45e-bf9bbc523424/volumes" Nov 02 20:00:15.822399 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:15.822380 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" path="/var/lib/kubelet/pods/58af3e1a-198c-4706-9bfa-e5b1ba476d4c/volumes" Nov 02 20:00:15.823300 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:15.823281 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" path="/var/lib/kubelet/pods/64524fe5-c084-40bd-9195-7ca010e99c99/volumes" Nov 02 20:00:16.808036 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:16.808005 5850 scope.go:122] "RemoveContainer" containerID="319fb0b14ab6aebff45fda3289f0518a294c9a368cb75a85ee9b5f1500aad325" Nov 02 20:00:16.808571 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:16.808179 5850 scope.go:122] "RemoveContainer" containerID="14617ab7a6cfe40a89964f89539ac5ed60dd68a6f09d137a5e4dd835d3b7a272" Nov 02 20:00:17.031413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:17.031323 5850 generic.go:356] "Generic (PLEG): container finished" podID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerID="32646cff6aa61e367d8ed9cdb456907e7262c34589c39a905c3f76dc2cde4d21" exitCode=60 Nov 02 20:00:17.031413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:17.031367 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" event={"ID":"c54d0bae-85b4-437d-bdae-9d2486ec98e1","Type":"ContainerDied","Data":"32646cff6aa61e367d8ed9cdb456907e7262c34589c39a905c3f76dc2cde4d21"} Nov 02 20:00:17.031682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:17.031434 5850 scope.go:122] "RemoveContainer" containerID="4ba8329a3db5ce3cad79e95084caf38e17a49124779e928aac101243eb4c9c71" Nov 02 20:00:17.031907 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:17.031886 5850 scope.go:122] "RemoveContainer" containerID="32646cff6aa61e367d8ed9cdb456907e7262c34589c39a905c3f76dc2cde4d21" Nov 02 20:00:17.032313 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:17.032289 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d_odh-ai-gateway-infra(c54d0bae-85b4-437d-bdae-9d2486ec98e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" Nov 02 20:00:18.039997 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.039910 5850 generic.go:356] "Generic (PLEG): container finished" podID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerID="b7171e0507654231fb4db7191521e3c1a6a3e36f6ed60c94c079c028f906309d" exitCode=60 Nov 02 20:00:18.040454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.039988 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" event={"ID":"1887850b-06fb-4dc3-b3e2-5ee558704129","Type":"ContainerDied","Data":"b7171e0507654231fb4db7191521e3c1a6a3e36f6ed60c94c079c028f906309d"} Nov 02 20:00:18.040454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.040055 5850 scope.go:122] "RemoveContainer" containerID="14617ab7a6cfe40a89964f89539ac5ed60dd68a6f09d137a5e4dd835d3b7a272" Nov 02 20:00:18.040570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.040510 5850 scope.go:122] "RemoveContainer" containerID="b7171e0507654231fb4db7191521e3c1a6a3e36f6ed60c94c079c028f906309d" Nov 02 20:00:18.040932 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:18.040905 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k_odh-ai-gateway-infra(1887850b-06fb-4dc3-b3e2-5ee558704129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" Nov 02 20:00:18.043396 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.043376 5850 generic.go:356] "Generic (PLEG): container finished" podID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerID="a00f18ccd08d3e1460f465ab203d3abb1f09443ad308f8f4d9e8cff82fb97e74" exitCode=60 Nov 02 20:00:18.043504 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.043447 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" event={"ID":"952f9f92-6ba2-4aac-93f1-a55cd17f98b0","Type":"ContainerDied","Data":"a00f18ccd08d3e1460f465ab203d3abb1f09443ad308f8f4d9e8cff82fb97e74"} Nov 02 20:00:18.043855 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.043841 5850 scope.go:122] "RemoveContainer" containerID="a00f18ccd08d3e1460f465ab203d3abb1f09443ad308f8f4d9e8cff82fb97e74" Nov 02 20:00:18.044229 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:18.044205 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr_odh-ai-gateway-infra(952f9f92-6ba2-4aac-93f1-a55cd17f98b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" Nov 02 20:00:18.053427 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.053407 5850 scope.go:122] "RemoveContainer" containerID="319fb0b14ab6aebff45fda3289f0518a294c9a368cb75a85ee9b5f1500aad325" Nov 02 20:00:18.157585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.157543 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d"] Nov 02 20:00:18.332461 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.332435 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:18.390997 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.390957 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54d0bae-85b4-437d-bdae-9d2486ec98e1-kube-api-access-h2c7j\" (UniqueName: \"kubernetes.io/projected/c54d0bae-85b4-437d-bdae-9d2486ec98e1-kube-api-access-h2c7j\") pod \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\" (UID: \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\") " Nov 02 20:00:18.391184 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.391125 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54d0bae-85b4-437d-bdae-9d2486ec98e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54d0bae-85b4-437d-bdae-9d2486ec98e1-maas-api-service-ca\") pod \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\" (UID: \"c54d0bae-85b4-437d-bdae-9d2486ec98e1\") " Nov 02 20:00:18.391848 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.391808 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54d0bae-85b4-437d-bdae-9d2486ec98e1-maas-api-service-ca" pod "c54d0bae-85b4-437d-bdae-9d2486ec98e1" (UID: "c54d0bae-85b4-437d-bdae-9d2486ec98e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:18.393422 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.393386 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54d0bae-85b4-437d-bdae-9d2486ec98e1-kube-api-access-h2c7j" pod "c54d0bae-85b4-437d-bdae-9d2486ec98e1" (UID: "c54d0bae-85b4-437d-bdae-9d2486ec98e1"). InnerVolumeSpecName "kube-api-access-h2c7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:18.493249 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.493210 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54d0bae-85b4-437d-bdae-9d2486ec98e1-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:18.493249 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:18.493251 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2c7j\" (UniqueName: \"kubernetes.io/projected/c54d0bae-85b4-437d-bdae-9d2486ec98e1-kube-api-access-h2c7j\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:19.009748 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.009717 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:00:19.053189 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.053161 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" Nov 02 20:00:19.053708 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.053190 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d" event={"ID":"c54d0bae-85b4-437d-bdae-9d2486ec98e1","Type":"ContainerDied","Data":"6c3889c48b357eb1c1b7b5d2714268645a612d4fdbce011a38348c257e016c98"} Nov 02 20:00:19.053708 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.053261 5850 scope.go:122] "RemoveContainer" containerID="32646cff6aa61e367d8ed9cdb456907e7262c34589c39a905c3f76dc2cde4d21" Nov 02 20:00:19.175637 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.175599 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k"] Nov 02 20:00:19.253346 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.253310 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d"] Nov 02 20:00:19.259049 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.258440 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8795bf-29894160-td68d"] Nov 02 20:00:19.289279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.287469 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr"] Nov 02 20:00:19.292894 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.292788 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457"] Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293552 5850 cpu_manager.go:414] "containerMap: removing container" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerName="cleanup" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293572 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerName="cleanup" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293585 5850 cpu_manager.go:414] "containerMap: removing container" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerName="cleanup" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293591 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerName="cleanup" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293620 5850 cpu_manager.go:414] "containerMap: removing container" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerName="cleanup" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293628 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerName="cleanup" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293650 5850 cpu_manager.go:414] "containerMap: removing container" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293655 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293672 5850 cpu_manager.go:414] "containerMap: removing container" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293676 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293704 5850 cpu_manager.go:414] "containerMap: removing container" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293708 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293735 5850 cpu_manager.go:414] "containerMap: removing container" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293739 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293882 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="44370a98-8413-404c-b45e-bf9bbc523424" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293893 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerName="cleanup" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293900 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerName="cleanup" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293920 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" containerName="cleanup" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293932 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="58af3e1a-198c-4706-9bfa-e5b1ba476d4c" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293939 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="64524fe5-c084-40bd-9195-7ca010e99c99" containerName="istio-proxy" Nov 02 20:00:19.293993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.293948 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="268a0dec-6781-41b8-b2c9-fe08f8d18ded" containerName="istio-proxy" Nov 02 20:00:19.299691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.299667 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.372820 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.372769 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ffdc4d-serving-cert\"" Nov 02 20:00:19.373583 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.373554 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457"] Nov 02 20:00:19.408184 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.407594 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.408184 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.407709 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:19.408184 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.407808 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e621b751-a8ef-4a02-914e-d0c6135e1337-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.408184 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.408041 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.408184 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.408096 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvbw9\" (UniqueName: \"kubernetes.io/projected/e621b751-a8ef-4a02-914e-d0c6135e1337-kube-api-access-zvbw9\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.508705 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.508676 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1887850b-06fb-4dc3-b3e2-5ee558704129-kube-api-access-2q8gb\" (UniqueName: \"kubernetes.io/projected/1887850b-06fb-4dc3-b3e2-5ee558704129-kube-api-access-2q8gb\") pod \"1887850b-06fb-4dc3-b3e2-5ee558704129\" (UID: \"1887850b-06fb-4dc3-b3e2-5ee558704129\") " Nov 02 20:00:19.508859 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.508836 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1887850b-06fb-4dc3-b3e2-5ee558704129-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1887850b-06fb-4dc3-b3e2-5ee558704129-maas-api-service-ca\") pod \"1887850b-06fb-4dc3-b3e2-5ee558704129\" (UID: \"1887850b-06fb-4dc3-b3e2-5ee558704129\") " Nov 02 20:00:19.509211 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.509186 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1887850b-06fb-4dc3-b3e2-5ee558704129-maas-api-service-ca" pod "1887850b-06fb-4dc3-b3e2-5ee558704129" (UID: "1887850b-06fb-4dc3-b3e2-5ee558704129"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:19.509312 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.509294 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.509466 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.509446 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbw9\" (UniqueName: \"kubernetes.io/projected/e621b751-a8ef-4a02-914e-d0c6135e1337-kube-api-access-zvbw9\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.509547 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:19.509466 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-serving-cert: secret "maas-api-e2e-shared-b-w3-ffdc4d-serving-cert" not found Nov 02 20:00:19.509605 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:19.509577 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls podName:e621b751-a8ef-4a02-914e-d0c6135e1337 nodeName:}" failed. No retries permitted until 2026-11-02 20:00:20.009554484 +0000 UTC m=+1878.351943548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls") pod "maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" (UID: "e621b751-a8ef-4a02-914e-d0c6135e1337") : secret "maas-api-e2e-shared-b-w3-ffdc4d-serving-cert" not found Nov 02 20:00:19.509660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.509614 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.509736 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.509718 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e621b751-a8ef-4a02-914e-d0c6135e1337-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.509928 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.509910 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1887850b-06fb-4dc3-b3e2-5ee558704129-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:19.510623 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.510599 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e621b751-a8ef-4a02-914e-d0c6135e1337-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.511367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.511340 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1887850b-06fb-4dc3-b3e2-5ee558704129-kube-api-access-2q8gb" pod "1887850b-06fb-4dc3-b3e2-5ee558704129" (UID: "1887850b-06fb-4dc3-b3e2-5ee558704129"). InnerVolumeSpecName "kube-api-access-2q8gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:19.512373 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.512241 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.547789 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.547710 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:19.561378 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.561343 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:19.568377 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.568347 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbw9\" (UniqueName: \"kubernetes.io/projected/e621b751-a8ef-4a02-914e-d0c6135e1337-kube-api-access-zvbw9\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:19.614495 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.613530 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-maas-api-service-ca\") pod \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\" (UID: \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\") " Nov 02 20:00:19.640996 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.619823 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-maas-api-service-ca" pod "952f9f92-6ba2-4aac-93f1-a55cd17f98b0" (UID: "952f9f92-6ba2-4aac-93f1-a55cd17f98b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:19.648843 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.646265 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-kube-api-access-c2pxx\" (UniqueName: \"kubernetes.io/projected/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-kube-api-access-c2pxx\") pod \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\" (UID: \"952f9f92-6ba2-4aac-93f1-a55cd17f98b0\") " Nov 02 20:00:19.648843 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.647351 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:19.648843 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.647380 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q8gb\" (UniqueName: \"kubernetes.io/projected/1887850b-06fb-4dc3-b3e2-5ee558704129-kube-api-access-2q8gb\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:19.652462 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.652424 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-kube-api-access-c2pxx" pod "952f9f92-6ba2-4aac-93f1-a55cd17f98b0" (UID: "952f9f92-6ba2-4aac-93f1-a55cd17f98b0"). InnerVolumeSpecName "kube-api-access-c2pxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:19.731095 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.731062 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:19.748947 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.748898 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2pxx\" (UniqueName: \"kubernetes.io/projected/952f9f92-6ba2-4aac-93f1-a55cd17f98b0-kube-api-access-c2pxx\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:19.837390 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.837295 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54d0bae-85b4-437d-bdae-9d2486ec98e1" path="/var/lib/kubelet/pods/c54d0bae-85b4-437d-bdae-9d2486ec98e1/volumes" Nov 02 20:00:19.894210 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:19.894159 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:20.071646 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.071610 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" Nov 02 20:00:20.072156 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.071644 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k" event={"ID":"1887850b-06fb-4dc3-b3e2-5ee558704129","Type":"ContainerDied","Data":"ae29a67acff6523bd6e968667ecc4b44205bca6cffa550e41986be84f0837295"} Nov 02 20:00:20.072156 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.071708 5850 scope.go:122] "RemoveContainer" containerID="b7171e0507654231fb4db7191521e3c1a6a3e36f6ed60c94c079c028f906309d" Nov 02 20:00:20.072156 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.072098 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:20.075335 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.075310 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:20.075804 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.075784 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" Nov 02 20:00:20.075895 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.075810 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr" event={"ID":"952f9f92-6ba2-4aac-93f1-a55cd17f98b0","Type":"ContainerDied","Data":"55ce507e9cedabd89a5283a1ae3273d5f693c81f212895c58ace2bf1d9f25909"} Nov 02 20:00:20.083953 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.083934 5850 scope.go:122] "RemoveContainer" containerID="a00f18ccd08d3e1460f465ab203d3abb1f09443ad308f8f4d9e8cff82fb97e74" Nov 02 20:00:20.217084 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.217046 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:20.241439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.241405 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr"] Nov 02 20:00:20.245279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.245239 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894160-k99xr"] Nov 02 20:00:20.368130 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.368095 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k"] Nov 02 20:00:20.375518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.375467 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894160-lgn5k"] Nov 02 20:00:20.596637 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:00:20.596600 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode621b751_a8ef_4a02_914e_d0c6135e1337.slice/crio-2ca043b0fde696b74621583bd8c035114a6acb08a86b031a29db99aba71950c3 WatchSource:0}: Error finding container 2ca043b0fde696b74621583bd8c035114a6acb08a86b031a29db99aba71950c3: Status 404 returned error can't find the container with id 2ca043b0fde696b74621583bd8c035114a6acb08a86b031a29db99aba71950c3 Nov 02 20:00:20.599411 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:20.599389 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457"] Nov 02 20:00:21.083552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:21.083506 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" event={"ID":"e621b751-a8ef-4a02-914e-d0c6135e1337","Type":"ContainerStarted","Data":"2ca043b0fde696b74621583bd8c035114a6acb08a86b031a29db99aba71950c3"} Nov 02 20:00:21.816613 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:21.816528 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" path="/var/lib/kubelet/pods/1887850b-06fb-4dc3-b3e2-5ee558704129/volumes" Nov 02 20:00:21.817088 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:21.817066 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" path="/var/lib/kubelet/pods/952f9f92-6ba2-4aac-93f1-a55cd17f98b0/volumes" Nov 02 20:00:22.093811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:22.093719 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" event={"ID":"e621b751-a8ef-4a02-914e-d0c6135e1337","Type":"ContainerStarted","Data":"783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e"} Nov 02 20:00:22.094220 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:22.093864 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:22.225501 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:22.225087 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" podStartSLOduration=2.3737491840000002 podStartE2EDuration="3.225066483s" totalImagesPullingTime="851.317299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:22.224347781 +0000 UTC m=+1880.566736862" watchObservedRunningTime="2026-11-02 20:00:22.225066483 +0000 UTC m=+1880.567455564" Nov 02 20:00:24.212650 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:24.212614 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:28.107368 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:28.107338 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:00:28.641309 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:28.641254 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:33.047516 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:33.047446 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:37.371693 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:37.371659 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:41.561613 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.561533 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv"] Nov 02 20:00:41.562227 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562212 5850 cpu_manager.go:414] "containerMap: removing container" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerName="cleanup" Nov 02 20:00:41.562273 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562228 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerName="cleanup" Nov 02 20:00:41.562273 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562262 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerName="cleanup" Nov 02 20:00:41.562273 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562268 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerName="cleanup" Nov 02 20:00:41.562360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562276 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerName="cleanup" Nov 02 20:00:41.562360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562282 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerName="cleanup" Nov 02 20:00:41.562360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562299 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerName="cleanup" Nov 02 20:00:41.562360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562305 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerName="cleanup" Nov 02 20:00:41.562360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562316 5850 cpu_manager.go:414] "containerMap: removing container" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerName="cleanup" Nov 02 20:00:41.562360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562320 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerName="cleanup" Nov 02 20:00:41.562612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562402 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerName="cleanup" Nov 02 20:00:41.562612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562414 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerName="cleanup" Nov 02 20:00:41.562612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562426 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerName="cleanup" Nov 02 20:00:41.562612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562439 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerName="cleanup" Nov 02 20:00:41.562612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.562449 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerName="cleanup" Nov 02 20:00:41.567602 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.567577 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.627550 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.627509 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv"] Nov 02 20:00:41.652307 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.652277 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4b0a12-serving-cert\"" Nov 02 20:00:41.688010 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.687973 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:41.744502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.739137 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.744502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.739242 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-metrics-certs\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.744502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.739562 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.744502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.739640 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47x8\" (UniqueName: \"kubernetes.io/projected/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-kube-api-access-q47x8\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.841094 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.841007 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.841094 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.841056 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47x8\" (UniqueName: \"kubernetes.io/projected/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-kube-api-access-q47x8\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.841285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.841140 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.841285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.841182 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-metrics-certs\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.841832 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.841808 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.843893 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.843874 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.843984 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.843942 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-metrics-certs\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.926552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.917246 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47x8\" (UniqueName: \"kubernetes.io/projected/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-kube-api-access-q47x8\") pod \"maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:41.955497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:41.951152 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:42.136328 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:42.136294 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:42.182200 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:42.182163 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:42.577055 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:00:42.577023 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4502b57c_0a04_448b_b2e5_90f9e9dd6e1b.slice/crio-9e42f81d9ec21fda54772ab9323465f9e66bfec71feac9a2cb6f0756031d9bbc WatchSource:0}: Error finding container 9e42f81d9ec21fda54772ab9323465f9e66bfec71feac9a2cb6f0756031d9bbc: Status 404 returned error can't find the container with id 9e42f81d9ec21fda54772ab9323465f9e66bfec71feac9a2cb6f0756031d9bbc Nov 02 20:00:42.579564 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:42.579541 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv"] Nov 02 20:00:43.258303 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:43.258268 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" event={"ID":"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b","Type":"ContainerStarted","Data":"9e42f81d9ec21fda54772ab9323465f9e66bfec71feac9a2cb6f0756031d9bbc"} Nov 02 20:00:43.280670 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:43.280617 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:44.268539 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:44.268498 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" event={"ID":"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b","Type":"ContainerStarted","Data":"40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3"} Nov 02 20:00:44.268998 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:44.268568 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:44.763356 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:00:44.763320 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:00:46.347606 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:46.347573 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:50.272286 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:50.272252 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:50.281462 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:50.281439 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:00:50.420832 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:50.420772 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" podStartSLOduration=8.611410561 podStartE2EDuration="9.420758039s" totalImagesPullingTime="809.347478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:44.391903833 +0000 UTC m=+1902.734292916" watchObservedRunningTime="2026-11-02 20:00:50.420758039 +0000 UTC m=+1908.763147119" Nov 02 20:00:50.430359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:50.430313 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:50.849847 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:50.849808 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:55.194351 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:00:55.194318 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:00.037394 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:00.037362 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:04.680011 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:04.679974 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:09.158055 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:09.158011 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:11.062163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:11.062124 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:11.228294 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:11.228256 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:13.530108 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:13.530046 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:14.186569 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:14.186521 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:14.761838 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:14.761801 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:14.763663 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:01:14.763638 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:01:17.947554 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:17.947512 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:22.523686 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:22.523649 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:27.479258 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:27.479219 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:28.495627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:28.495588 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:28.655948 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:28.655912 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:31.675394 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.675354 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67"] Nov 02 20:01:31.676206 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.676174 5850 cpu_manager.go:414] "containerMap: removing container" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerName="cleanup" Nov 02 20:01:31.676206 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.676201 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952f9f92-6ba2-4aac-93f1-a55cd17f98b0" containerName="cleanup" Nov 02 20:01:31.676412 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.676382 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1887850b-06fb-4dc3-b3e2-5ee558704129" containerName="cleanup" Nov 02 20:01:31.680197 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.680177 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.724576 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.724534 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.724758 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.724688 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8vbl\" (UniqueName: \"kubernetes.io/projected/2f6b3f54-dc0c-420c-9007-caa095fc3185-kube-api-access-d8vbl\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.724758 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.724724 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-metrics-certs\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.724838 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.724775 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b3f54-dc0c-420c-9007-caa095fc3185-openshift-service-ca\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.737384 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.737347 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67"] Nov 02 20:01:31.784535 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.784498 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7ab11604-serving-cert\"" Nov 02 20:01:31.826928 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.826889 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.827103 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:01:31.827025 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-serving-cert: secret "maas-api-e2e-mt-7ab11604-serving-cert" not found Nov 02 20:01:31.827103 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.827033 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8vbl\" (UniqueName: \"kubernetes.io/projected/2f6b3f54-dc0c-420c-9007-caa095fc3185-kube-api-access-d8vbl\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.827103 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.827069 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-metrics-certs\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.827103 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:01:31.827087 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls podName:2f6b3f54-dc0c-420c-9007-caa095fc3185 nodeName:}" failed. No retries permitted until 2026-11-02 20:01:32.327069885 +0000 UTC m=+1950.669458943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls") pod "maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" (UID: "2f6b3f54-dc0c-420c-9007-caa095fc3185") : secret "maas-api-e2e-mt-7ab11604-serving-cert" not found Nov 02 20:01:31.827413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.827396 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b3f54-dc0c-420c-9007-caa095fc3185-openshift-service-ca\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.827996 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.827970 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b3f54-dc0c-420c-9007-caa095fc3185-openshift-service-ca\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.829935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.829911 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-metrics-certs\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:31.869409 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.869370 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:31.887720 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:31.887686 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8vbl\" (UniqueName: \"kubernetes.io/projected/2f6b3f54-dc0c-420c-9007-caa095fc3185-kube-api-access-d8vbl\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:32.030012 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:32.029927 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:32.179866 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:32.179824 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:32.345176 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:32.345076 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:32.347704 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:32.347674 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls\") pod \"maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:32.595413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:32.595319 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:32.968614 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:01:32.968581 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6b3f54_dc0c_420c_9007_caa095fc3185.slice/crio-373566b1d3cdf6d5317b087d9116877eb070f09e097fc982304d9c1cc3c8ab54 WatchSource:0}: Error finding container 373566b1d3cdf6d5317b087d9116877eb070f09e097fc982304d9c1cc3c8ab54: Status 404 returned error can't find the container with id 373566b1d3cdf6d5317b087d9116877eb070f09e097fc982304d9c1cc3c8ab54 Nov 02 20:01:32.970395 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:32.970373 5850 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:01:32.971559 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:32.971513 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67"] Nov 02 20:01:33.843089 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:33.843054 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" event={"ID":"2f6b3f54-dc0c-420c-9007-caa095fc3185","Type":"ContainerStarted","Data":"373566b1d3cdf6d5317b087d9116877eb070f09e097fc982304d9c1cc3c8ab54"} Nov 02 20:01:34.852507 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:34.852423 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" event={"ID":"2f6b3f54-dc0c-420c-9007-caa095fc3185","Type":"ContainerStarted","Data":"f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1"} Nov 02 20:01:34.852958 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:34.852538 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:34.972056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:34.972000 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" podStartSLOduration=3.048566701 podStartE2EDuration="3.971985533s" totalImagesPullingTime="923.418832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:34.969229958 +0000 UTC m=+1953.311619038" watchObservedRunningTime="2026-11-02 20:01:34.971985533 +0000 UTC m=+1953.314374613" Nov 02 20:01:36.457978 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:36.457944 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:39.027765 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:39.027723 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx"] Nov 02 20:01:39.028245 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:39.027948 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" containerID="cri-o://9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2" gracePeriod=30 Nov 02 20:01:40.867265 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:40.867236 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:01:41.016002 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:41.015968 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:43.288639 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:43.288587 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:01:43.289040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:43.288693 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:01:44.311572 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.311542 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 20:01:44.404633 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.404577 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-token\") pod \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " Nov 02 20:01:44.404852 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.404665 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-data\") pod \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " Nov 02 20:01:44.404852 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.404690 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-envoy\") pod \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " Nov 02 20:01:44.404852 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.404719 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-kube-api-access-dtktr\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-kube-api-access-dtktr\") pod \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " Nov 02 20:01:44.404852 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.404760 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-credential-socket\") pod \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " Nov 02 20:01:44.404852 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.404842 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istiod-ca-cert\") pod \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " Nov 02 20:01:44.405135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.404893 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-certs\") pod \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " Nov 02 20:01:44.405135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.404936 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-podinfo\") pod \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " Nov 02 20:01:44.405135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405038 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-data" pod "1bfd26c4-7283-49e5-bbd6-d69ff19bce71" (UID: "1bfd26c4-7283-49e5-bbd6-d69ff19bce71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.405300 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405153 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-socket\") pod \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\" (UID: \"1bfd26c4-7283-49e5-bbd6-d69ff19bce71\") " Nov 02 20:01:44.405300 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405203 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-certs" pod "1bfd26c4-7283-49e5-bbd6-d69ff19bce71" (UID: "1bfd26c4-7283-49e5-bbd6-d69ff19bce71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.405300 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405258 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-credential-socket" pod "1bfd26c4-7283-49e5-bbd6-d69ff19bce71" (UID: "1bfd26c4-7283-49e5-bbd6-d69ff19bce71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.405300 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405282 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istiod-ca-cert" pod "1bfd26c4-7283-49e5-bbd6-d69ff19bce71" (UID: "1bfd26c4-7283-49e5-bbd6-d69ff19bce71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:44.405584 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405561 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-socket" pod "1bfd26c4-7283-49e5-bbd6-d69ff19bce71" (UID: "1bfd26c4-7283-49e5-bbd6-d69ff19bce71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.405880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405864 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.405963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405880 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.405963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405891 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.405963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405900 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.405963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.405910 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.407182 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.407148 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-envoy" pod "1bfd26c4-7283-49e5-bbd6-d69ff19bce71" (UID: "1bfd26c4-7283-49e5-bbd6-d69ff19bce71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.407314 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.407193 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-token" pod "1bfd26c4-7283-49e5-bbd6-d69ff19bce71" (UID: "1bfd26c4-7283-49e5-bbd6-d69ff19bce71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:44.407448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.407430 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-kube-api-access-dtktr" pod "1bfd26c4-7283-49e5-bbd6-d69ff19bce71" (UID: "1bfd26c4-7283-49e5-bbd6-d69ff19bce71"). InnerVolumeSpecName "kube-api-access-dtktr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:44.407594 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.407572 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-podinfo" pod "1bfd26c4-7283-49e5-bbd6-d69ff19bce71" (UID: "1bfd26c4-7283-49e5-bbd6-d69ff19bce71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:01:44.507148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.507111 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.507148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.507146 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.507372 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.507157 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtktr\" (UniqueName: \"kubernetes.io/projected/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-kube-api-access-dtktr\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.507372 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.507167 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfd26c4-7283-49e5-bbd6-d69ff19bce71-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.763561 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:01:44.763516 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:01:44.941327 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.941182 5850 generic.go:356] "Generic (PLEG): container finished" podID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerID="9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2" exitCode=0 Nov 02 20:01:44.941327 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.941264 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" Nov 02 20:01:44.941327 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.941298 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" event={"ID":"1bfd26c4-7283-49e5-bbd6-d69ff19bce71","Type":"ContainerDied","Data":"9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2"} Nov 02 20:01:44.941669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.941380 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx" event={"ID":"1bfd26c4-7283-49e5-bbd6-d69ff19bce71","Type":"ContainerDied","Data":"36375854f3e1d26edf4089c65b39372555bcea10a1242e0f721dbd36fc681039"} Nov 02 20:01:44.941669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.941407 5850 scope.go:122] "RemoveContainer" containerID="9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2" Nov 02 20:01:44.962179 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.961949 5850 scope.go:122] "RemoveContainer" containerID="9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2" Nov 02 20:01:44.962584 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:01:44.962464 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2\": container with ID starting with 9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2 not found: ID does not exist" containerID="9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2" Nov 02 20:01:44.962584 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:44.962554 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2"} err="failed to get container status \"9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2\": rpc error: code = NotFound desc = could not find container \"9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2\": container with ID starting with 9fb3d479fa7262c3fe6c1e57faefdeadf3da433f3ad2373b99301620dc2fefb2 not found: ID does not exist" Nov 02 20:01:45.163168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:45.163129 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx"] Nov 02 20:01:45.214203 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:45.214165 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-45zvx"] Nov 02 20:01:45.672301 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:45.672270 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:45.820339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:45.820291 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" path="/var/lib/kubelet/pods/1bfd26c4-7283-49e5-bbd6-d69ff19bce71/volumes" Nov 02 20:01:45.823878 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:45.823839 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:45.984143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:45.984059 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:50.814500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:50.806286 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:53.855422 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:53.855383 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn"] Nov 02 20:01:53.969279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:53.969233 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp"] Nov 02 20:01:54.256221 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:54.256185 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:54.422332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:54.422296 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:55.451239 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:55.451185 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:58.855900 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:58.855844 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" podUID="b0005d91-7470-4ec7-9087-a53710e3e94c" containerName="payload-processing" containerID="cri-o://0c2499eff28e65ecaecb38e30f7dbd2f7d40c92c41e911f63b0b11ca7ce34ec8" gracePeriod=25 Nov 02 20:01:58.970256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:58.970191 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" podUID="d9e71fad-4636-4a1f-a2e4-a5bd787d7eba" containerName="payload-pre-processing" containerID="cri-o://bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18" gracePeriod=25 Nov 02 20:01:59.060244 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.060195 5850 generic.go:356] "Generic (PLEG): container finished" podID="b0005d91-7470-4ec7-9087-a53710e3e94c" containerID="0c2499eff28e65ecaecb38e30f7dbd2f7d40c92c41e911f63b0b11ca7ce34ec8" exitCode=0 Nov 02 20:01:59.060525 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.060446 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" event={"ID":"b0005d91-7470-4ec7-9087-a53710e3e94c","Type":"ContainerDied","Data":"0c2499eff28e65ecaecb38e30f7dbd2f7d40c92c41e911f63b0b11ca7ce34ec8"} Nov 02 20:01:59.165419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.165382 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 20:01:59.258525 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.258501 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 20:01:59.291799 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.291762 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0005d91-7470-4ec7-9087-a53710e3e94c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0005d91-7470-4ec7-9087-a53710e3e94c-plugins-config-volume\") pod \"b0005d91-7470-4ec7-9087-a53710e3e94c\" (UID: \"b0005d91-7470-4ec7-9087-a53710e3e94c\") " Nov 02 20:01:59.291994 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.291859 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0005d91-7470-4ec7-9087-a53710e3e94c-kube-api-access-mz46k\" (UniqueName: \"kubernetes.io/projected/b0005d91-7470-4ec7-9087-a53710e3e94c-kube-api-access-mz46k\") pod \"b0005d91-7470-4ec7-9087-a53710e3e94c\" (UID: \"b0005d91-7470-4ec7-9087-a53710e3e94c\") " Nov 02 20:01:59.292217 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.292191 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0005d91-7470-4ec7-9087-a53710e3e94c-plugins-config-volume" pod "b0005d91-7470-4ec7-9087-a53710e3e94c" (UID: "b0005d91-7470-4ec7-9087-a53710e3e94c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:59.292573 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.292554 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0005d91-7470-4ec7-9087-a53710e3e94c-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:59.294201 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.294178 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0005d91-7470-4ec7-9087-a53710e3e94c-kube-api-access-mz46k" pod "b0005d91-7470-4ec7-9087-a53710e3e94c" (UID: "b0005d91-7470-4ec7-9087-a53710e3e94c"). InnerVolumeSpecName "kube-api-access-mz46k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:59.394162 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.394060 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-kube-api-access-4khk7\" (UniqueName: \"kubernetes.io/projected/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-kube-api-access-4khk7\") pod \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\" (UID: \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\") " Nov 02 20:01:59.394335 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.394315 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-plugins-config-volume\") pod \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\" (UID: \"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba\") " Nov 02 20:01:59.394691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.394664 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-plugins-config-volume" pod "d9e71fad-4636-4a1f-a2e4-a5bd787d7eba" (UID: "d9e71fad-4636-4a1f-a2e4-a5bd787d7eba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:59.394810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.394786 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:59.394810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.394805 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz46k\" (UniqueName: \"kubernetes.io/projected/b0005d91-7470-4ec7-9087-a53710e3e94c-kube-api-access-mz46k\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:59.396189 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.396168 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-kube-api-access-4khk7" pod "d9e71fad-4636-4a1f-a2e4-a5bd787d7eba" (UID: "d9e71fad-4636-4a1f-a2e4-a5bd787d7eba"). InnerVolumeSpecName "kube-api-access-4khk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:59.496580 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:01:59.496543 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4khk7\" (UniqueName: \"kubernetes.io/projected/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba-kube-api-access-4khk7\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:00.027941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.027909 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:00.107626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.095346 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" Nov 02 20:02:00.107626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.096786 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn" event={"ID":"b0005d91-7470-4ec7-9087-a53710e3e94c","Type":"ContainerDied","Data":"e3b3957228fb4c2d79006ceb8c575847fe225706c3aa18195b0cb9d39873c388"} Nov 02 20:02:00.107626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.096858 5850 scope.go:122] "RemoveContainer" containerID="0c2499eff28e65ecaecb38e30f7dbd2f7d40c92c41e911f63b0b11ca7ce34ec8" Nov 02 20:02:00.111311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.109704 5850 generic.go:356] "Generic (PLEG): container finished" podID="d9e71fad-4636-4a1f-a2e4-a5bd787d7eba" containerID="bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18" exitCode=0 Nov 02 20:02:00.111311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.109787 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" event={"ID":"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba","Type":"ContainerDied","Data":"bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18"} Nov 02 20:02:00.111311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.109866 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" event={"ID":"d9e71fad-4636-4a1f-a2e4-a5bd787d7eba","Type":"ContainerDied","Data":"3ef10960a5673dc689c7f9b72ba3eaeece50b19bf1133b1e22b626a21180e19d"} Nov 02 20:02:00.111311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.110090 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp" Nov 02 20:02:00.127613 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.127588 5850 scope.go:122] "RemoveContainer" containerID="bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18" Nov 02 20:02:00.138943 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.138920 5850 scope.go:122] "RemoveContainer" containerID="bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18" Nov 02 20:02:00.139262 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:02:00.139237 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18\": container with ID starting with bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18 not found: ID does not exist" containerID="bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18" Nov 02 20:02:00.139331 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.139272 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18"} err="failed to get container status \"bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18\": rpc error: code = NotFound desc = could not find container \"bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18\": container with ID starting with bb621132e4a6318c5f65b442d1816c454e1595251706ddc7af31846dd46d8e18 not found: ID does not exist" Nov 02 20:02:00.270233 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.270179 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn"] Nov 02 20:02:00.318327 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.318285 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b7eed-6df547859c-nsrnn"] Nov 02 20:02:00.390896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.390863 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp"] Nov 02 20:02:00.397021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:00.396992 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b7eed-6b79cc65c9-gf2xp"] Nov 02 20:02:01.761437 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:01.761388 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:01.903614 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:01.903569 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0005d91-7470-4ec7-9087-a53710e3e94c" path="/var/lib/kubelet/pods/b0005d91-7470-4ec7-9087-a53710e3e94c/volumes" Nov 02 20:02:01.904631 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:01.904602 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e71fad-4636-4a1f-a2e4-a5bd787d7eba" path="/var/lib/kubelet/pods/d9e71fad-4636-4a1f-a2e4-a5bd787d7eba/volumes" Nov 02 20:02:01.929871 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:01.929838 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:02.087757 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:02.087677 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:04.520639 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:04.520606 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:09.349496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:09.349260 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:14.283966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:14.283931 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:14.494944 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:14.494900 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:14.763430 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:02:14.763386 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:02:19.038748 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:19.038711 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:22.099017 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:22.098973 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:22.258841 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:22.258782 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:23.577488 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:23.577390 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:27.921314 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:27.921281 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:32.419655 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:32.419614 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:36.847141 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:36.847111 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:41.549962 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:41.549930 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:44.762403 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:02:44.762360 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:02:45.357430 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.357390 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl"] Nov 02 20:02:45.357712 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.357687 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" podUID="75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" containerName="maas-api" containerID="cri-o://60bcecb5ba5fd464c65c0422d60680e442eba16ddd7f7ca11b7f3161824eb4bf" gracePeriod=30 Nov 02 20:02:45.473216 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.473176 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:45.655517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.655441 5850 generic.go:356] "Generic (PLEG): container finished" podID="75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" containerID="60bcecb5ba5fd464c65c0422d60680e442eba16ddd7f7ca11b7f3161824eb4bf" exitCode=0 Nov 02 20:02:45.655684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.655565 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" event={"ID":"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f","Type":"ContainerDied","Data":"60bcecb5ba5fd464c65c0422d60680e442eba16ddd7f7ca11b7f3161824eb4bf"} Nov 02 20:02:45.742983 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.742595 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 20:02:45.882379 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.882332 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-metrics-certs\") pod \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " Nov 02 20:02:45.884698 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.884666 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-openshift-service-ca\") pod \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " Nov 02 20:02:45.884855 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.884763 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-kube-api-access-ksvtd\" (UniqueName: \"kubernetes.io/projected/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-kube-api-access-ksvtd\") pod \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " Nov 02 20:02:45.884919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.884858 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls\") pod \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\" (UID: \"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f\") " Nov 02 20:02:45.885299 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.885110 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-openshift-service-ca" pod "75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" (UID: "75ef5acc-2c2f-4d74-9aa2-721859e4ac1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:02:45.885400 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.885290 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-metrics-certs" pod "75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" (UID: "75ef5acc-2c2f-4d74-9aa2-721859e4ac1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:02:45.885633 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.885609 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:45.885752 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.885637 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:45.887536 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.887509 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls" pod "75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" (UID: "75ef5acc-2c2f-4d74-9aa2-721859e4ac1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:02:45.887765 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.887744 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-kube-api-access-ksvtd" pod "75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" (UID: "75ef5acc-2c2f-4d74-9aa2-721859e4ac1f"). InnerVolumeSpecName "kube-api-access-ksvtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:02:45.990895 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.990794 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvtd\" (UniqueName: \"kubernetes.io/projected/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-kube-api-access-ksvtd\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:45.990895 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:45.990834 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:46.663696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:46.663665 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" Nov 02 20:02:46.663902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:46.663700 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl" event={"ID":"75ef5acc-2c2f-4d74-9aa2-721859e4ac1f","Type":"ContainerDied","Data":"5f7f3c5850f54522f4831fc9b5635df6b65b4f1285ae53b4f403e673b961645d"} Nov 02 20:02:46.663902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:46.663762 5850 scope.go:122] "RemoveContainer" containerID="60bcecb5ba5fd464c65c0422d60680e442eba16ddd7f7ca11b7f3161824eb4bf" Nov 02 20:02:46.834196 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:46.834155 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl"] Nov 02 20:02:46.885348 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:46.885303 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8479ff-7c69447f88-zfhsl"] Nov 02 20:02:46.923163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:46.923076 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:47.177905 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:47.177815 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:47.324247 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:47.324217 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:47.817545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:47.817511 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" path="/var/lib/kubelet/pods/75ef5acc-2c2f-4d74-9aa2-721859e4ac1f/volumes" Nov 02 20:02:47.923448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:47.923410 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:51.802139 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:51.802099 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:53.642319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:53.642274 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:53.859693 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:53.859661 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:54.038010 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:54.037257 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:56.267656 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:02:56.267622 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:00.924168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:00.924127 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:05.542308 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:05.542256 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:07.995829 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:07.995782 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:08.151619 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:08.151584 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:09.310351 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:09.310322 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:10.444164 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:10.444130 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2"] Nov 02 20:03:10.732244 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:10.732149 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:10.887654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:10.887621 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:11.046577 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:11.046435 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:11.637419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:11.637385 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:13.399252 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:13.399207 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:13.590556 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:13.588708 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:14.900500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:14.896299 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:14.989887 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:03:14.989641 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:03:15.007221 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.007188 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67"] Nov 02 20:03:15.007767 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.007734 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" podUID="2f6b3f54-dc0c-420c-9007-caa095fc3185" containerName="maas-api" containerID="cri-o://f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1" gracePeriod=30 Nov 02 20:03:15.495713 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.477520 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" podUID="237eb2b7-58b5-4acc-97c2-084473f92f0c" containerName="payload-pre-processing" containerID="cri-o://7d283c4398ad4330b28d76a7dd8aab51776bddd06c302afc0a1a865e68dbb69e" gracePeriod=25 Nov 02 20:03:15.692020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.691988 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:03:15.790263 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.790161 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6b3f54-dc0c-420c-9007-caa095fc3185-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b3f54-dc0c-420c-9007-caa095fc3185-openshift-service-ca\") pod \"2f6b3f54-dc0c-420c-9007-caa095fc3185\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " Nov 02 20:03:15.790444 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.790367 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6b3f54-dc0c-420c-9007-caa095fc3185-kube-api-access-d8vbl\" (UniqueName: \"kubernetes.io/projected/2f6b3f54-dc0c-420c-9007-caa095fc3185-kube-api-access-d8vbl\") pod \"2f6b3f54-dc0c-420c-9007-caa095fc3185\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " Nov 02 20:03:15.790444 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.790416 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls\") pod \"2f6b3f54-dc0c-420c-9007-caa095fc3185\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " Nov 02 20:03:15.790594 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.790571 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-metrics-certs\") pod \"2f6b3f54-dc0c-420c-9007-caa095fc3185\" (UID: \"2f6b3f54-dc0c-420c-9007-caa095fc3185\") " Nov 02 20:03:15.790762 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.790662 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6b3f54-dc0c-420c-9007-caa095fc3185-openshift-service-ca" pod "2f6b3f54-dc0c-420c-9007-caa095fc3185" (UID: "2f6b3f54-dc0c-420c-9007-caa095fc3185"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:15.791102 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.791078 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b3f54-dc0c-420c-9007-caa095fc3185-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:15.793618 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.793460 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-metrics-certs" pod "2f6b3f54-dc0c-420c-9007-caa095fc3185" (UID: "2f6b3f54-dc0c-420c-9007-caa095fc3185"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:15.794655 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.794629 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls" pod "2f6b3f54-dc0c-420c-9007-caa095fc3185" (UID: "2f6b3f54-dc0c-420c-9007-caa095fc3185"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:15.794655 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.794642 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6b3f54-dc0c-420c-9007-caa095fc3185-kube-api-access-d8vbl" pod "2f6b3f54-dc0c-420c-9007-caa095fc3185" (UID: "2f6b3f54-dc0c-420c-9007-caa095fc3185"). InnerVolumeSpecName "kube-api-access-d8vbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:15.892331 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.892295 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:15.892331 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.892328 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8vbl\" (UniqueName: \"kubernetes.io/projected/2f6b3f54-dc0c-420c-9007-caa095fc3185-kube-api-access-d8vbl\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:15.892331 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.892340 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f6b3f54-dc0c-420c-9007-caa095fc3185-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:15.905926 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.905894 5850 generic.go:356] "Generic (PLEG): container finished" podID="2f6b3f54-dc0c-420c-9007-caa095fc3185" containerID="f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1" exitCode=0 Nov 02 20:03:15.906367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.906023 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" Nov 02 20:03:15.906367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.906034 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" event={"ID":"2f6b3f54-dc0c-420c-9007-caa095fc3185","Type":"ContainerDied","Data":"f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1"} Nov 02 20:03:15.906367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.906094 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67" event={"ID":"2f6b3f54-dc0c-420c-9007-caa095fc3185","Type":"ContainerDied","Data":"373566b1d3cdf6d5317b087d9116877eb070f09e097fc982304d9c1cc3c8ab54"} Nov 02 20:03:15.906367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.906122 5850 scope.go:122] "RemoveContainer" containerID="f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1" Nov 02 20:03:15.948621 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.948571 5850 generic.go:356] "Generic (PLEG): container finished" podID="237eb2b7-58b5-4acc-97c2-084473f92f0c" containerID="7d283c4398ad4330b28d76a7dd8aab51776bddd06c302afc0a1a865e68dbb69e" exitCode=0 Nov 02 20:03:15.948783 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.948642 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" event={"ID":"237eb2b7-58b5-4acc-97c2-084473f92f0c","Type":"ContainerDied","Data":"7d283c4398ad4330b28d76a7dd8aab51776bddd06c302afc0a1a865e68dbb69e"} Nov 02 20:03:15.953350 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.953323 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 20:03:15.962468 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.962442 5850 scope.go:122] "RemoveContainer" containerID="f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1" Nov 02 20:03:15.962876 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:03:15.962849 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1\": container with ID starting with f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1 not found: ID does not exist" containerID="f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1" Nov 02 20:03:15.962951 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:15.962889 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1"} err="failed to get container status \"f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1\": rpc error: code = NotFound desc = could not find container \"f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1\": container with ID starting with f87d36299cb5334b024c80a600fc862ec7e750612cd1b3b78791117b562f43b1 not found: ID does not exist" Nov 02 20:03:16.083136 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.082926 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67"] Nov 02 20:03:16.087051 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.086937 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ab11604-8459b5dfdd-h8j67"] Nov 02 20:03:16.094970 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.094940 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237eb2b7-58b5-4acc-97c2-084473f92f0c-kube-api-access-cf7qb\" (UniqueName: \"kubernetes.io/projected/237eb2b7-58b5-4acc-97c2-084473f92f0c-kube-api-access-cf7qb\") pod \"237eb2b7-58b5-4acc-97c2-084473f92f0c\" (UID: \"237eb2b7-58b5-4acc-97c2-084473f92f0c\") " Nov 02 20:03:16.095193 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.095177 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237eb2b7-58b5-4acc-97c2-084473f92f0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237eb2b7-58b5-4acc-97c2-084473f92f0c-plugins-config-volume\") pod \"237eb2b7-58b5-4acc-97c2-084473f92f0c\" (UID: \"237eb2b7-58b5-4acc-97c2-084473f92f0c\") " Nov 02 20:03:16.095568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.095546 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237eb2b7-58b5-4acc-97c2-084473f92f0c-plugins-config-volume" pod "237eb2b7-58b5-4acc-97c2-084473f92f0c" (UID: "237eb2b7-58b5-4acc-97c2-084473f92f0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:16.095719 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.095703 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237eb2b7-58b5-4acc-97c2-084473f92f0c-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:16.097131 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.097108 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237eb2b7-58b5-4acc-97c2-084473f92f0c-kube-api-access-cf7qb" pod "237eb2b7-58b5-4acc-97c2-084473f92f0c" (UID: "237eb2b7-58b5-4acc-97c2-084473f92f0c"). InnerVolumeSpecName "kube-api-access-cf7qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:16.196887 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.196849 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf7qb\" (UniqueName: \"kubernetes.io/projected/237eb2b7-58b5-4acc-97c2-084473f92f0c-kube-api-access-cf7qb\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:16.982019 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.981986 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" Nov 02 20:03:16.982707 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.982063 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2" event={"ID":"237eb2b7-58b5-4acc-97c2-084473f92f0c","Type":"ContainerDied","Data":"0568e7c6c9823335079301d6b7b1bd6fd053b99179823c883e585e93aec562bc"} Nov 02 20:03:16.982859 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:16.982743 5850 scope.go:122] "RemoveContainer" containerID="7d283c4398ad4330b28d76a7dd8aab51776bddd06c302afc0a1a865e68dbb69e" Nov 02 20:03:17.063763 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:17.063723 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:17.154627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:17.154456 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2"] Nov 02 20:03:17.205663 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:17.205626 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4fd0d5-65c67fdfc5-9rcb2"] Nov 02 20:03:17.497336 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:17.497300 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:17.655923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:17.655875 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:17.813429 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:17.813321 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:17.817359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:17.817332 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237eb2b7-58b5-4acc-97c2-084473f92f0c" path="/var/lib/kubelet/pods/237eb2b7-58b5-4acc-97c2-084473f92f0c/volumes" Nov 02 20:03:17.818050 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:17.818027 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6b3f54-dc0c-420c-9007-caa095fc3185" path="/var/lib/kubelet/pods/2f6b3f54-dc0c-420c-9007-caa095fc3185/volumes" Nov 02 20:03:18.078295 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:18.078161 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:20.076406 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:20.076370 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:21.016263 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.016224 5850 generic.go:356] "Generic (PLEG): container finished" podID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerID="6d84bc8142d363c4ae6aaf1879076088ce79afa56a8ca07241ed90782672fe69" exitCode=0 Nov 02 20:03:21.016545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.016289 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" event={"ID":"1a00dd58-66dc-4d13-97b8-acfcf52513e9","Type":"ContainerDied","Data":"6d84bc8142d363c4ae6aaf1879076088ce79afa56a8ca07241ed90782672fe69"} Nov 02 20:03:21.016878 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.016855 5850 scope.go:122] "RemoveContainer" containerID="6d84bc8142d363c4ae6aaf1879076088ce79afa56a8ca07241ed90782672fe69" Nov 02 20:03:21.019041 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.019001 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:03:21.019201 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.019098 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:03:21.019201 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.019145 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:03:21.035889 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.035835 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:21.583546 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.583452 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:21.797762 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.797727 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 20:03:21.827726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:21.827686 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:22.042661 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:22.042625 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" event={"ID":"1a00dd58-66dc-4d13-97b8-acfcf52513e9","Type":"ContainerStarted","Data":"7d730f888944ca3479e706e2579ab81810458bbe86fe565a7f88b8de044bfa8a"} Nov 02 20:03:22.151027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:22.150996 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:22.566113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:22.566071 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:22.753419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:22.753381 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:23.038938 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:23.038901 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 20:03:23.040113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:23.040079 5850 patch_prober.go:29] interesting pod/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 20:03:23.040259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:23.040145 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 20:03:23.384682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:23.384643 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:23.561272 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:23.561236 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:23.696377 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:23.696285 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27"] Nov 02 20:03:23.725161 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:23.725097 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:23.895378 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:23.895346 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:24.044938 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:24.044838 5850 patch_prober.go:29] interesting pod/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 20:03:24.045217 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:24.045173 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 20:03:24.140404 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:24.140357 5850 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" secret="" err="secret \"e2e-mt-7ab11604-openshift-default-dockercfg-t7lsd\" not found" Nov 02 20:03:24.185927 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:24.185899 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:24.345945 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:24.345853 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:24.786708 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:24.786673 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:25.040063 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:25.039965 5850 patch_prober.go:29] interesting pod/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 20:03:25.040063 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:25.040039 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 20:03:25.148620 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:25.148578 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" containerID="cri-o://7d730f888944ca3479e706e2579ab81810458bbe86fe565a7f88b8de044bfa8a" gracePeriod=30 Nov 02 20:03:25.954146 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:25.954081 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:26.164163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:26.164124 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:26.331311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:26.331219 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:28.972174 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:28.972142 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:29.202277 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:29.202242 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:29.364721 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:29.364637 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:29.565359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:29.565325 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:30.765711 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:30.765675 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:30.967170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:30.967127 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:31.050129 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:31.050039 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l"] Nov 02 20:03:31.050347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:31.050319 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" containerID="cri-o://d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb" gracePeriod=30 Nov 02 20:03:31.178616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:31.160458 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:31.553836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:31.553791 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:32.188499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:32.171540 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:32.405970 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:32.405747 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:33.088164 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.088129 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg"] Nov 02 20:03:33.088387 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.088334 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" podUID="1d9b5dd6-c428-432f-a1db-b60e6aecb665" containerName="maas-api" containerID="cri-o://01f1e99d1f88cacb579ef7dbd837f74328c972774e28ded3e6422fa42c9d0185" gracePeriod=30 Nov 02 20:03:33.206513 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.206469 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:33.386425 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.386384 5850 generic.go:356] "Generic (PLEG): container finished" podID="1d9b5dd6-c428-432f-a1db-b60e6aecb665" containerID="01f1e99d1f88cacb579ef7dbd837f74328c972774e28ded3e6422fa42c9d0185" exitCode=0 Nov 02 20:03:33.386664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.386643 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" event={"ID":"1d9b5dd6-c428-432f-a1db-b60e6aecb665","Type":"ContainerDied","Data":"01f1e99d1f88cacb579ef7dbd837f74328c972774e28ded3e6422fa42c9d0185"} Nov 02 20:03:33.389391 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.389360 5850 generic.go:356] "Generic (PLEG): container finished" podID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerID="7d730f888944ca3479e706e2579ab81810458bbe86fe565a7f88b8de044bfa8a" exitCode=0 Nov 02 20:03:33.389585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.389489 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" event={"ID":"1a00dd58-66dc-4d13-97b8-acfcf52513e9","Type":"ContainerDied","Data":"7d730f888944ca3479e706e2579ab81810458bbe86fe565a7f88b8de044bfa8a"} Nov 02 20:03:33.389585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.389540 5850 scope.go:122] "RemoveContainer" containerID="6d84bc8142d363c4ae6aaf1879076088ce79afa56a8ca07241ed90782672fe69" Nov 02 20:03:33.603779 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.593292 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 20:03:33.611773 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.611744 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 20:03:33.712562 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.712465 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-certs\") pod \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " Nov 02 20:03:33.712736 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.712571 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istiod-ca-cert\") pod \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " Nov 02 20:03:33.712736 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.712592 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-credential-socket\") pod \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " Nov 02 20:03:33.712736 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.712627 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-podinfo\") pod \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " Nov 02 20:03:33.712736 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.712652 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-metrics-certs\") pod \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " Nov 02 20:03:33.713106 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.712773 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-envoy\") pod \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " Nov 02 20:03:33.713106 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.712801 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-certs" pod "1a00dd58-66dc-4d13-97b8-acfcf52513e9" (UID: "1a00dd58-66dc-4d13-97b8-acfcf52513e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:33.713106 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.712939 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9b5dd6-c428-432f-a1db-b60e6aecb665-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b5dd6-c428-432f-a1db-b60e6aecb665-openshift-service-ca\") pod \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " Nov 02 20:03:33.714298 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.714270 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-data\") pod \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " Nov 02 20:03:33.714408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.714321 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9b5dd6-c428-432f-a1db-b60e6aecb665-kube-api-access-58blk\" (UniqueName: \"kubernetes.io/projected/1d9b5dd6-c428-432f-a1db-b60e6aecb665-kube-api-access-58blk\") pod \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " Nov 02 20:03:33.714408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.714358 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-socket\") pod \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " Nov 02 20:03:33.714573 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.714423 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-kube-api-access-hg95h\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-kube-api-access-hg95h\") pod \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " Nov 02 20:03:33.714573 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.714508 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-token\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-token\") pod \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\" (UID: \"1a00dd58-66dc-4d13-97b8-acfcf52513e9\") " Nov 02 20:03:33.714573 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.714563 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls\") pod \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\" (UID: \"1d9b5dd6-c428-432f-a1db-b60e6aecb665\") " Nov 02 20:03:33.715366 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.715321 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.716168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.713041 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istiod-ca-cert" pod "1a00dd58-66dc-4d13-97b8-acfcf52513e9" (UID: "1a00dd58-66dc-4d13-97b8-acfcf52513e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:33.716168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.713536 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-credential-socket" pod "1a00dd58-66dc-4d13-97b8-acfcf52513e9" (UID: "1a00dd58-66dc-4d13-97b8-acfcf52513e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:33.716168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.714140 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9b5dd6-c428-432f-a1db-b60e6aecb665-openshift-service-ca" pod "1d9b5dd6-c428-432f-a1db-b60e6aecb665" (UID: "1d9b5dd6-c428-432f-a1db-b60e6aecb665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:33.716168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.715509 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-metrics-certs" pod "1d9b5dd6-c428-432f-a1db-b60e6aecb665" (UID: "1d9b5dd6-c428-432f-a1db-b60e6aecb665"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:33.716168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.716152 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-envoy" pod "1a00dd58-66dc-4d13-97b8-acfcf52513e9" (UID: "1a00dd58-66dc-4d13-97b8-acfcf52513e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:33.717592 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.716420 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-podinfo" pod "1a00dd58-66dc-4d13-97b8-acfcf52513e9" (UID: "1a00dd58-66dc-4d13-97b8-acfcf52513e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:03:33.717592 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.716495 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-data" pod "1a00dd58-66dc-4d13-97b8-acfcf52513e9" (UID: "1a00dd58-66dc-4d13-97b8-acfcf52513e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:33.717978 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.717952 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-socket" pod "1a00dd58-66dc-4d13-97b8-acfcf52513e9" (UID: "1a00dd58-66dc-4d13-97b8-acfcf52513e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:33.718366 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.718339 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9b5dd6-c428-432f-a1db-b60e6aecb665-kube-api-access-58blk" pod "1d9b5dd6-c428-432f-a1db-b60e6aecb665" (UID: "1d9b5dd6-c428-432f-a1db-b60e6aecb665"). InnerVolumeSpecName "kube-api-access-58blk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:33.718464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.718435 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls" pod "1d9b5dd6-c428-432f-a1db-b60e6aecb665" (UID: "1d9b5dd6-c428-432f-a1db-b60e6aecb665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:33.718579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.718533 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-kube-api-access-hg95h" pod "1a00dd58-66dc-4d13-97b8-acfcf52513e9" (UID: "1a00dd58-66dc-4d13-97b8-acfcf52513e9"). InnerVolumeSpecName "kube-api-access-hg95h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:33.719822 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.719795 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-token" pod "1a00dd58-66dc-4d13-97b8-acfcf52513e9" (UID: "1a00dd58-66dc-4d13-97b8-acfcf52513e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:33.816891 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.816855 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.816891 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.816893 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.816908 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.816924 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.816937 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.816952 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b5dd6-c428-432f-a1db-b60e6aecb665-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.816965 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.816979 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58blk\" (UniqueName: \"kubernetes.io/projected/1d9b5dd6-c428-432f-a1db-b60e6aecb665-kube-api-access-58blk\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.816996 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a00dd58-66dc-4d13-97b8-acfcf52513e9-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.817010 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg95h\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-kube-api-access-hg95h\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.817026 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a00dd58-66dc-4d13-97b8-acfcf52513e9-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:33.817097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:33.817040 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9b5dd6-c428-432f-a1db-b60e6aecb665-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:34.397915 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.397884 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" Nov 02 20:03:34.397915 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.397900 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg" event={"ID":"1d9b5dd6-c428-432f-a1db-b60e6aecb665","Type":"ContainerDied","Data":"a7f3d8b9cdd34e4455632858d8178d84ee22e7012243b827d26d9da9c62e3fd4"} Nov 02 20:03:34.398496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.397984 5850 scope.go:122] "RemoveContainer" containerID="01f1e99d1f88cacb579ef7dbd837f74328c972774e28ded3e6422fa42c9d0185" Nov 02 20:03:34.399794 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.399769 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" event={"ID":"1a00dd58-66dc-4d13-97b8-acfcf52513e9","Type":"ContainerDied","Data":"813271b8648310d72dcd3f0f7348ff25a97f98713ae1bdbdd24b282841455f7d"} Nov 02 20:03:34.399794 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.399786 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27" Nov 02 20:03:34.408469 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.408448 5850 scope.go:122] "RemoveContainer" containerID="7d730f888944ca3479e706e2579ab81810458bbe86fe565a7f88b8de044bfa8a" Nov 02 20:03:34.568685 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.568646 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:34.569629 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.569605 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27"] Nov 02 20:03:34.620736 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.620702 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7ab11604-openshift-default-66b8f56b65-h6b27"] Nov 02 20:03:34.682389 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.682285 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg"] Nov 02 20:03:34.685595 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.685565 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-4fd0d5-6fd4cc7bfb-v5dvg"] Nov 02 20:03:34.818209 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:34.818175 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:35.131549 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.117883 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.136989 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc"] Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138045 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138071 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138124 5850 cpu_manager.go:414] "containerMap: removing container" podUID="b0005d91-7470-4ec7-9087-a53710e3e94c" containerName="payload-processing" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138135 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0005d91-7470-4ec7-9087-a53710e3e94c" containerName="payload-processing" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138153 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138161 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138200 5850 cpu_manager.go:414] "containerMap: removing container" podUID="237eb2b7-58b5-4acc-97c2-084473f92f0c" containerName="payload-pre-processing" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138207 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237eb2b7-58b5-4acc-97c2-084473f92f0c" containerName="payload-pre-processing" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138220 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1d9b5dd6-c428-432f-a1db-b60e6aecb665" containerName="maas-api" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138227 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9b5dd6-c428-432f-a1db-b60e6aecb665" containerName="maas-api" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138240 5850 cpu_manager.go:414] "containerMap: removing container" podUID="2f6b3f54-dc0c-420c-9007-caa095fc3185" containerName="maas-api" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138248 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6b3f54-dc0c-420c-9007-caa095fc3185" containerName="maas-api" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138260 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138268 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138281 5850 cpu_manager.go:414] "containerMap: removing container" podUID="75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" containerName="maas-api" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138288 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" containerName="maas-api" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138302 5850 cpu_manager.go:414] "containerMap: removing container" podUID="d9e71fad-4636-4a1f-a2e4-a5bd787d7eba" containerName="payload-pre-processing" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138310 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e71fad-4636-4a1f-a2e4-a5bd787d7eba" containerName="payload-pre-processing" Nov 02 20:03:35.140502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138454 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0005d91-7470-4ec7-9087-a53710e3e94c" containerName="payload-processing" Nov 02 20:03:35.167560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.138466 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" Nov 02 20:03:35.167560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.166835 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="237eb2b7-58b5-4acc-97c2-084473f92f0c" containerName="payload-pre-processing" Nov 02 20:03:35.167560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.166859 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9b5dd6-c428-432f-a1db-b60e6aecb665" containerName="maas-api" Nov 02 20:03:35.167560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.166875 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6b3f54-dc0c-420c-9007-caa095fc3185" containerName="maas-api" Nov 02 20:03:35.167560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.166889 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfd26c4-7283-49e5-bbd6-d69ff19bce71" containerName="istio-proxy" Nov 02 20:03:35.167560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.166901 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ef5acc-2c2f-4d74-9aa2-721859e4ac1f" containerName="maas-api" Nov 02 20:03:35.167560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.166914 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e71fad-4636-4a1f-a2e4-a5bd787d7eba" containerName="payload-pre-processing" Nov 02 20:03:35.337505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.337438 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.346633 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.346602 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:35.355037 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.354996 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc"] Nov 02 20:03:35.396506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.396408 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7025fa-openshift-default-dockercfg-zmbhs\"" Nov 02 20:03:35.402539 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.402507 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-token\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.403021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.402589 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-podinfo\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.403021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.402668 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-envoy\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.403021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.402731 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwcj6\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-kube-api-access-cwcj6\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.403021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.402776 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-credential-socket\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.403021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.402813 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-certs\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.403021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.402970 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-socket\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.403021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.402998 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istiod-ca-cert\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.403326 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.403033 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-data\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.507637 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-token\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.507731 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-podinfo\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.507819 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-envoy\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.507851 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwcj6\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-kube-api-access-cwcj6\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.507896 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-credential-socket\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.507935 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-certs\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.508149 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-socket\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.508177 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istiod-ca-cert\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.508211 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-data\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.509030 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-data\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.510349 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-credential-socket\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.511513 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-socket\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.512176 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-certs\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.513496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.512586 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istiod-ca-cert\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.531504 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.529100 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:35.542947 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.542900 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-envoy\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.558129 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.558096 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-podinfo\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.568046 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.568008 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-token\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.569360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.569330 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwcj6\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-kube-api-access-cwcj6\") pod \"e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.661790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.661701 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:35.715588 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.715080 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:35.816364 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.816329 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" path="/var/lib/kubelet/pods/1a00dd58-66dc-4d13-97b8-acfcf52513e9/volumes" Nov 02 20:03:35.816857 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:35.816842 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9b5dd6-c428-432f-a1db-b60e6aecb665" path="/var/lib/kubelet/pods/1d9b5dd6-c428-432f-a1db-b60e6aecb665/volumes" Nov 02 20:03:36.089611 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.089578 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc"] Nov 02 20:03:36.116887 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:03:36.116848 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda632b5f5_8af8_4ec1_9d4d_7970df12d254.slice/crio-3f6dec780a81fdc3eaaeea8e4abf8556a889a7f83c870c1ca23fbcf493b2a5bb WatchSource:0}: Error finding container 3f6dec780a81fdc3eaaeea8e4abf8556a889a7f83c870c1ca23fbcf493b2a5bb: Status 404 returned error can't find the container with id 3f6dec780a81fdc3eaaeea8e4abf8556a889a7f83c870c1ca23fbcf493b2a5bb Nov 02 20:03:36.156846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.156802 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:03:36.156954 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.156879 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:03:36.156954 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.156907 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:03:36.334761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.334734 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 20:03:36.418810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.418775 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" event={"ID":"a632b5f5-8af8-4ec1-9d4d-7970df12d254","Type":"ContainerStarted","Data":"e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6"} Nov 02 20:03:36.419313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.418840 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" event={"ID":"a632b5f5-8af8-4ec1-9d4d-7970df12d254","Type":"ContainerStarted","Data":"3f6dec780a81fdc3eaaeea8e4abf8556a889a7f83c870c1ca23fbcf493b2a5bb"} Nov 02 20:03:36.420266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420240 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-socket\") pod \"aa77d63a-7ab5-4e82-9144-15039098659b\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " Nov 02 20:03:36.420404 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420342 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa77d63a-7ab5-4e82-9144-15039098659b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa77d63a-7ab5-4e82-9144-15039098659b-istio-podinfo\") pod \"aa77d63a-7ab5-4e82-9144-15039098659b\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " Nov 02 20:03:36.420404 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420383 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-istio-token\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-istio-token\") pod \"aa77d63a-7ab5-4e82-9144-15039098659b\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " Nov 02 20:03:36.420558 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420439 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-credential-socket\") pod \"aa77d63a-7ab5-4e82-9144-15039098659b\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " Nov 02 20:03:36.420643 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420563 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-socket" pod "aa77d63a-7ab5-4e82-9144-15039098659b" (UID: "aa77d63a-7ab5-4e82-9144-15039098659b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:36.420748 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420641 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-kube-api-access-7vc9p\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-kube-api-access-7vc9p\") pod \"aa77d63a-7ab5-4e82-9144-15039098659b\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " Nov 02 20:03:36.420748 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420716 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa77d63a-7ab5-4e82-9144-15039098659b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa77d63a-7ab5-4e82-9144-15039098659b-istiod-ca-cert\") pod \"aa77d63a-7ab5-4e82-9144-15039098659b\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " Nov 02 20:03:36.420872 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420797 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-certs\") pod \"aa77d63a-7ab5-4e82-9144-15039098659b\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " Nov 02 20:03:36.420872 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420821 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-envoy\") pod \"aa77d63a-7ab5-4e82-9144-15039098659b\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " Nov 02 20:03:36.420872 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420840 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-data\") pod \"aa77d63a-7ab5-4e82-9144-15039098659b\" (UID: \"aa77d63a-7ab5-4e82-9144-15039098659b\") " Nov 02 20:03:36.421026 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.420903 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-credential-socket" pod "aa77d63a-7ab5-4e82-9144-15039098659b" (UID: "aa77d63a-7ab5-4e82-9144-15039098659b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:36.421135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.421113 5850 generic.go:356] "Generic (PLEG): container finished" podID="aa77d63a-7ab5-4e82-9144-15039098659b" containerID="d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb" exitCode=0 Nov 02 20:03:36.421370 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.421199 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" Nov 02 20:03:36.421643 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.421414 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" event={"ID":"aa77d63a-7ab5-4e82-9144-15039098659b","Type":"ContainerDied","Data":"d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb"} Nov 02 20:03:36.421841 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.421814 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l" event={"ID":"aa77d63a-7ab5-4e82-9144-15039098659b","Type":"ContainerDied","Data":"d0ee92fa1e08dd1d6aec2acac319da66bf1182e9894c48a22829d318b9b6b0e6"} Nov 02 20:03:36.422042 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.421670 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-data" pod "aa77d63a-7ab5-4e82-9144-15039098659b" (UID: "aa77d63a-7ab5-4e82-9144-15039098659b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:36.422042 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.421430 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.422225 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.422045 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.422225 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.421754 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa77d63a-7ab5-4e82-9144-15039098659b-istiod-ca-cert" pod "aa77d63a-7ab5-4e82-9144-15039098659b" (UID: "aa77d63a-7ab5-4e82-9144-15039098659b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:36.422225 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.421813 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-certs" pod "aa77d63a-7ab5-4e82-9144-15039098659b" (UID: "aa77d63a-7ab5-4e82-9144-15039098659b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:36.422225 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.421850 5850 scope.go:122] "RemoveContainer" containerID="d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb" Nov 02 20:03:36.423184 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.423161 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa77d63a-7ab5-4e82-9144-15039098659b-istio-podinfo" pod "aa77d63a-7ab5-4e82-9144-15039098659b" (UID: "aa77d63a-7ab5-4e82-9144-15039098659b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:03:36.423283 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.423194 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-istio-token" pod "aa77d63a-7ab5-4e82-9144-15039098659b" (UID: "aa77d63a-7ab5-4e82-9144-15039098659b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:36.423283 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.423276 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-kube-api-access-7vc9p" pod "aa77d63a-7ab5-4e82-9144-15039098659b" (UID: "aa77d63a-7ab5-4e82-9144-15039098659b"). InnerVolumeSpecName "kube-api-access-7vc9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:36.427668 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.427641 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-envoy" pod "aa77d63a-7ab5-4e82-9144-15039098659b" (UID: "aa77d63a-7ab5-4e82-9144-15039098659b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:36.432944 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.432920 5850 scope.go:122] "RemoveContainer" containerID="d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc" Nov 02 20:03:36.443622 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.443597 5850 scope.go:122] "RemoveContainer" containerID="d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb" Nov 02 20:03:36.444027 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:03:36.443989 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb\": container with ID starting with d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb not found: ID does not exist" containerID="d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb" Nov 02 20:03:36.444141 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.444036 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb"} err="failed to get container status \"d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb\": rpc error: code = NotFound desc = could not find container \"d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb\": container with ID starting with d8d66818ffb561686ab9f15441f85bdede18a3e16e27b327904ebc2297b84afb not found: ID does not exist" Nov 02 20:03:36.444141 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.444063 5850 scope.go:122] "RemoveContainer" containerID="d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc" Nov 02 20:03:36.444351 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:03:36.444333 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc\": container with ID starting with d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc not found: ID does not exist" containerID="d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc" Nov 02 20:03:36.444393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.444362 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc"} err="failed to get container status \"d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc\": rpc error: code = NotFound desc = could not find container \"d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc\": container with ID starting with d0f6da5b256252ad69949be0ce81e72e2ce9753e76c03022490006c7bd75e9dc not found: ID does not exist" Nov 02 20:03:36.523310 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.523270 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vc9p\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-kube-api-access-7vc9p\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.523310 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.523311 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa77d63a-7ab5-4e82-9144-15039098659b-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.523310 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.523327 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.523618 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.523343 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.523618 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.523357 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa77d63a-7ab5-4e82-9144-15039098659b-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.523618 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.523369 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa77d63a-7ab5-4e82-9144-15039098659b-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.523618 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.523383 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa77d63a-7ab5-4e82-9144-15039098659b-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.630870 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.630813 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" podStartSLOduration=1.630796418 podStartE2EDuration="1.630796418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:36.628248503 +0000 UTC m=+2074.970637587" watchObservedRunningTime="2026-11-02 20:03:36.630796418 +0000 UTC m=+2074.973185498" Nov 02 20:03:36.662879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.662832 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:36.853143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.853100 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l"] Nov 02 20:03:36.857139 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.857108 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89a22a-openshift-default-6dfbfd7667-zpt8l"] Nov 02 20:03:36.881916 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:36.881824 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:37.048720 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:37.048686 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:37.293855 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:37.293771 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:37.668297 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:37.668265 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:37.816553 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:37.816519 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" path="/var/lib/kubelet/pods/aa77d63a-7ab5-4e82-9144-15039098659b/volumes" Nov 02 20:03:38.437232 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:38.437201 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:38.438378 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:38.438358 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:03:39.788763 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:39.788666 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:40.132906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:40.132872 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:40.317790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:40.317587 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:40.502312 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:40.502225 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:42.065235 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:42.065190 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:42.252499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:42.238769 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:42.409464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:42.409423 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:44.159605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:44.159570 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:44.763300 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:03:44.763258 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:03:45.722615 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:45.722578 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:45.915197 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:45.892354 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:46.081647 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:46.081561 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:46.579208 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:46.579169 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:49.503997 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:49.503952 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:49.725113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:49.725053 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:50.928887 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:50.928847 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:52.362841 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:52.362805 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:52.525373 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:52.525337 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:52.692518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:52.692458 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m"] Nov 02 20:03:52.692756 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:52.692727 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" containerID="cri-o://fef0fc9919cc9fe80619b5db859024d3517c592f304993f9ceae8b46ba6e755a" gracePeriod=30 Nov 02 20:03:52.706031 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:52.705994 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:55.776024 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:55.775990 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:57.331685 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:57.331650 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:57.498854 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:57.498773 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:57.841347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:57.841312 5850 generic.go:356] "Generic (PLEG): container finished" podID="05973ae4-c92f-460b-a550-76b094ea96fd" containerID="fef0fc9919cc9fe80619b5db859024d3517c592f304993f9ceae8b46ba6e755a" exitCode=0 Nov 02 20:03:57.841546 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:57.841358 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" event={"ID":"05973ae4-c92f-460b-a550-76b094ea96fd","Type":"ContainerDied","Data":"fef0fc9919cc9fe80619b5db859024d3517c592f304993f9ceae8b46ba6e755a"} Nov 02 20:03:57.841546 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:57.841422 5850 scope.go:122] "RemoveContainer" containerID="e0d9484c96f5f1a4530576e5c01b2e102173787f8d64fe2d924fdf731b80bb71" Nov 02 20:03:57.988775 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:57.988741 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 20:03:58.135447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135407 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-credential-socket\") pod \"05973ae4-c92f-460b-a550-76b094ea96fd\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " Nov 02 20:03:58.135684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135454 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-kube-api-access-8cxq9\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-kube-api-access-8cxq9\") pod \"05973ae4-c92f-460b-a550-76b094ea96fd\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " Nov 02 20:03:58.135684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135614 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-certs\") pod \"05973ae4-c92f-460b-a550-76b094ea96fd\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " Nov 02 20:03:58.135684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135660 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-socket\") pod \"05973ae4-c92f-460b-a550-76b094ea96fd\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " Nov 02 20:03:58.135851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135702 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-envoy\") pod \"05973ae4-c92f-460b-a550-76b094ea96fd\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " Nov 02 20:03:58.135851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135748 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-credential-socket" pod "05973ae4-c92f-460b-a550-76b094ea96fd" (UID: "05973ae4-c92f-460b-a550-76b094ea96fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:58.135851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135775 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-istio-token\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-istio-token\") pod \"05973ae4-c92f-460b-a550-76b094ea96fd\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " Nov 02 20:03:58.136005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135851 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-data\") pod \"05973ae4-c92f-460b-a550-76b094ea96fd\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " Nov 02 20:03:58.136005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135853 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-certs" pod "05973ae4-c92f-460b-a550-76b094ea96fd" (UID: "05973ae4-c92f-460b-a550-76b094ea96fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:58.136005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135899 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05973ae4-c92f-460b-a550-76b094ea96fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05973ae4-c92f-460b-a550-76b094ea96fd-istio-podinfo\") pod \"05973ae4-c92f-460b-a550-76b094ea96fd\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " Nov 02 20:03:58.136005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135913 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-socket" pod "05973ae4-c92f-460b-a550-76b094ea96fd" (UID: "05973ae4-c92f-460b-a550-76b094ea96fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:58.136005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.135939 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05973ae4-c92f-460b-a550-76b094ea96fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05973ae4-c92f-460b-a550-76b094ea96fd-istiod-ca-cert\") pod \"05973ae4-c92f-460b-a550-76b094ea96fd\" (UID: \"05973ae4-c92f-460b-a550-76b094ea96fd\") " Nov 02 20:03:58.136252 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.136069 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-data" pod "05973ae4-c92f-460b-a550-76b094ea96fd" (UID: "05973ae4-c92f-460b-a550-76b094ea96fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:58.136390 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.136364 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05973ae4-c92f-460b-a550-76b094ea96fd-istiod-ca-cert" pod "05973ae4-c92f-460b-a550-76b094ea96fd" (UID: "05973ae4-c92f-460b-a550-76b094ea96fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:58.136618 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.136600 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:58.136724 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.136622 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05973ae4-c92f-460b-a550-76b094ea96fd-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:58.136724 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.136638 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:58.136724 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.136653 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:58.136724 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.136666 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:58.137988 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.137964 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-envoy" pod "05973ae4-c92f-460b-a550-76b094ea96fd" (UID: "05973ae4-c92f-460b-a550-76b094ea96fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:58.138105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.138036 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-kube-api-access-8cxq9" pod "05973ae4-c92f-460b-a550-76b094ea96fd" (UID: "05973ae4-c92f-460b-a550-76b094ea96fd"). InnerVolumeSpecName "kube-api-access-8cxq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:58.138313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.138291 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-istio-token" pod "05973ae4-c92f-460b-a550-76b094ea96fd" (UID: "05973ae4-c92f-460b-a550-76b094ea96fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:58.138372 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.138324 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05973ae4-c92f-460b-a550-76b094ea96fd-istio-podinfo" pod "05973ae4-c92f-460b-a550-76b094ea96fd" (UID: "05973ae4-c92f-460b-a550-76b094ea96fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:03:58.238173 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.238094 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cxq9\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-kube-api-access-8cxq9\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:58.238173 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.238126 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05973ae4-c92f-460b-a550-76b094ea96fd-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:58.238173 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.238136 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05973ae4-c92f-460b-a550-76b094ea96fd-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:58.238173 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.238145 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05973ae4-c92f-460b-a550-76b094ea96fd-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:58.889091 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.888314 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" event={"ID":"05973ae4-c92f-460b-a550-76b094ea96fd","Type":"ContainerDied","Data":"3735fe521f1b8cbaab9093af947a6464963ac44ba5a97cc092e57dcc515aef35"} Nov 02 20:03:58.889091 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.888398 5850 scope.go:122] "RemoveContainer" containerID="fef0fc9919cc9fe80619b5db859024d3517c592f304993f9ceae8b46ba6e755a" Nov 02 20:03:58.889091 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:58.888558 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m" Nov 02 20:03:59.113978 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:59.113939 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m"] Nov 02 20:03:59.162414 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:59.162324 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b0cc24-openshift-default-64b6fdcc67-s7q7m"] Nov 02 20:03:59.816022 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:03:59.815988 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" path="/var/lib/kubelet/pods/05973ae4-c92f-460b-a550-76b094ea96fd/volumes" Nov 02 20:04:00.572853 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:00.572821 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:02.161136 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:02.161091 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:04.967767 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:04.967731 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:05.144687 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:05.144582 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:05.301578 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:05.301050 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:09.335346 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:09.335299 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:13.773761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:13.773718 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:14.762677 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:04:14.762632 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:04:15.196332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:15.196299 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:15.732923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:15.732868 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:18.143718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:18.143653 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:22.539437 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:22.539385 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:26.921061 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:26.921016 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:31.261343 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:31.261301 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:35.638639 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:35.638599 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:38.350805 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:38.350766 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:38.515192 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:38.515156 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:39.287024 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.286984 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj"] Nov 02 20:04:39.287232 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.287196 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" podUID="c204339b-7ab7-43b1-afff-456ac421b761" containerName="maas-api" containerID="cri-o://c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13" gracePeriod=30 Nov 02 20:04:39.404965 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.404914 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:39.623519 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.623456 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 20:04:39.668966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.668920 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls\") pod \"c204339b-7ab7-43b1-afff-456ac421b761\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " Nov 02 20:04:39.669124 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.669016 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c204339b-7ab7-43b1-afff-456ac421b761-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c204339b-7ab7-43b1-afff-456ac421b761-openshift-service-ca\") pod \"c204339b-7ab7-43b1-afff-456ac421b761\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " Nov 02 20:04:39.669124 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.669080 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-metrics-certs\") pod \"c204339b-7ab7-43b1-afff-456ac421b761\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " Nov 02 20:04:39.669259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.669171 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c204339b-7ab7-43b1-afff-456ac421b761-kube-api-access-r7v9n\" (UniqueName: \"kubernetes.io/projected/c204339b-7ab7-43b1-afff-456ac421b761-kube-api-access-r7v9n\") pod \"c204339b-7ab7-43b1-afff-456ac421b761\" (UID: \"c204339b-7ab7-43b1-afff-456ac421b761\") " Nov 02 20:04:39.669393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.669365 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c204339b-7ab7-43b1-afff-456ac421b761-openshift-service-ca" pod "c204339b-7ab7-43b1-afff-456ac421b761" (UID: "c204339b-7ab7-43b1-afff-456ac421b761"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:39.669743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.669715 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c204339b-7ab7-43b1-afff-456ac421b761-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:39.671176 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.671153 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls" pod "c204339b-7ab7-43b1-afff-456ac421b761" (UID: "c204339b-7ab7-43b1-afff-456ac421b761"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:39.671286 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.671262 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-metrics-certs" pod "c204339b-7ab7-43b1-afff-456ac421b761" (UID: "c204339b-7ab7-43b1-afff-456ac421b761"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:39.671336 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.671298 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c204339b-7ab7-43b1-afff-456ac421b761-kube-api-access-r7v9n" pod "c204339b-7ab7-43b1-afff-456ac421b761" (UID: "c204339b-7ab7-43b1-afff-456ac421b761"). InnerVolumeSpecName "kube-api-access-r7v9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:39.790191 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.785346 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7v9n\" (UniqueName: \"kubernetes.io/projected/c204339b-7ab7-43b1-afff-456ac421b761-kube-api-access-r7v9n\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:39.790607 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.790462 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:39.790607 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:39.790582 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c204339b-7ab7-43b1-afff-456ac421b761-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:40.244004 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.243913 5850 generic.go:356] "Generic (PLEG): container finished" podID="c204339b-7ab7-43b1-afff-456ac421b761" containerID="c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13" exitCode=0 Nov 02 20:04:40.244612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.244580 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" event={"ID":"c204339b-7ab7-43b1-afff-456ac421b761","Type":"ContainerDied","Data":"c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13"} Nov 02 20:04:40.244870 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.244847 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" event={"ID":"c204339b-7ab7-43b1-afff-456ac421b761","Type":"ContainerDied","Data":"687ab93c82be0eff4957bfc0135c2072429ece7f5852740e56390fd127be3879"} Nov 02 20:04:40.245167 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.245119 5850 scope.go:122] "RemoveContainer" containerID="c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13" Nov 02 20:04:40.245627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.245606 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj" Nov 02 20:04:40.333152 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.333119 5850 scope.go:122] "RemoveContainer" containerID="c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13" Nov 02 20:04:40.333576 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:04:40.333547 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13\": container with ID starting with c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13 not found: ID does not exist" containerID="c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13" Nov 02 20:04:40.333688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.333590 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13"} err="failed to get container status \"c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13\": rpc error: code = NotFound desc = could not find container \"c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13\": container with ID starting with c433107ff7ddea5b19f9ab384a9fb8cb68b0f233a27ea9a356f2ea145b910f13 not found: ID does not exist" Nov 02 20:04:40.442191 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.442139 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj"] Nov 02 20:04:40.493700 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.493661 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b7eed-dc7469cf4-zsfkj"] Nov 02 20:04:40.796523 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:40.796456 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:41.026731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:41.026700 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:41.196502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:41.189110 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:41.368285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:41.368252 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:41.726370 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:41.726339 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:41.816673 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:41.816641 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c204339b-7ab7-43b1-afff-456ac421b761" path="/var/lib/kubelet/pods/c204339b-7ab7-43b1-afff-456ac421b761/volumes" Nov 02 20:04:44.762859 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:04:44.762819 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:04:45.910099 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:45.910064 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:47.513717 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:47.513686 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:47.803890 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:47.803776 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:47.842042 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:47.842007 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g"] Nov 02 20:04:47.842253 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:47.842229 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" containerID="cri-o://21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650" gracePeriod=30 Nov 02 20:04:47.980904 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:47.980867 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:50.566571 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:50.566519 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:52.798653 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:52.798622 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:52.954127 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:52.954098 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:53.111884 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.111856 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 20:04:53.132101 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.132054 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-istio-token\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-istio-token\") pod \"69514c97-be51-45d0-818f-29c4d2e69318\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " Nov 02 20:04:53.137064 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.135134 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-istio-token" pod "69514c97-be51-45d0-818f-29c4d2e69318" (UID: "69514c97-be51-45d0-818f-29c4d2e69318"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:53.233605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.233566 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69514c97-be51-45d0-818f-29c4d2e69318-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69514c97-be51-45d0-818f-29c4d2e69318-istiod-ca-cert\") pod \"69514c97-be51-45d0-818f-29c4d2e69318\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " Nov 02 20:04:53.233788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.233620 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69514c97-be51-45d0-818f-29c4d2e69318-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69514c97-be51-45d0-818f-29c4d2e69318-istio-podinfo\") pod \"69514c97-be51-45d0-818f-29c4d2e69318\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " Nov 02 20:04:53.233788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.233654 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-socket\") pod \"69514c97-be51-45d0-818f-29c4d2e69318\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " Nov 02 20:04:53.233788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.233683 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-credential-socket\") pod \"69514c97-be51-45d0-818f-29c4d2e69318\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " Nov 02 20:04:53.233788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.233759 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-envoy\") pod \"69514c97-be51-45d0-818f-29c4d2e69318\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " Nov 02 20:04:53.233995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.233945 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-socket" pod "69514c97-be51-45d0-818f-29c4d2e69318" (UID: "69514c97-be51-45d0-818f-29c4d2e69318"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:04:53.234040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.233993 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-data\") pod \"69514c97-be51-45d0-818f-29c4d2e69318\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " Nov 02 20:04:53.234040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.233995 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-credential-socket" pod "69514c97-be51-45d0-818f-29c4d2e69318" (UID: "69514c97-be51-45d0-818f-29c4d2e69318"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:04:53.234040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234033 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69514c97-be51-45d0-818f-29c4d2e69318-istiod-ca-cert" pod "69514c97-be51-45d0-818f-29c4d2e69318" (UID: "69514c97-be51-45d0-818f-29c4d2e69318"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:53.234220 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234116 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-certs\") pod \"69514c97-be51-45d0-818f-29c4d2e69318\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " Nov 02 20:04:53.234220 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234146 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-kube-api-access-qjdpq\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-kube-api-access-qjdpq\") pod \"69514c97-be51-45d0-818f-29c4d2e69318\" (UID: \"69514c97-be51-45d0-818f-29c4d2e69318\") " Nov 02 20:04:53.234341 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234309 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-data" pod "69514c97-be51-45d0-818f-29c4d2e69318" (UID: "69514c97-be51-45d0-818f-29c4d2e69318"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:04:53.234460 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234417 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-certs" pod "69514c97-be51-45d0-818f-29c4d2e69318" (UID: "69514c97-be51-45d0-818f-29c4d2e69318"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:04:53.234779 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234752 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:53.234896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234781 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:53.234896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234797 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69514c97-be51-45d0-818f-29c4d2e69318-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:53.234896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234812 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:53.234896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234827 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:53.234896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.234841 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:53.236105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.236077 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69514c97-be51-45d0-818f-29c4d2e69318-istio-podinfo" pod "69514c97-be51-45d0-818f-29c4d2e69318" (UID: "69514c97-be51-45d0-818f-29c4d2e69318"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:04:53.236210 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.236159 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-envoy" pod "69514c97-be51-45d0-818f-29c4d2e69318" (UID: "69514c97-be51-45d0-818f-29c4d2e69318"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:04:53.236273 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.236255 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-kube-api-access-qjdpq" pod "69514c97-be51-45d0-818f-29c4d2e69318" (UID: "69514c97-be51-45d0-818f-29c4d2e69318"). InnerVolumeSpecName "kube-api-access-qjdpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:53.336503 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.336393 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjdpq\" (UniqueName: \"kubernetes.io/projected/69514c97-be51-45d0-818f-29c4d2e69318-kube-api-access-qjdpq\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:53.336503 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.336428 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69514c97-be51-45d0-818f-29c4d2e69318-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:53.336503 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.336442 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69514c97-be51-45d0-818f-29c4d2e69318-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:53.337974 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.337945 5850 generic.go:356] "Generic (PLEG): container finished" podID="69514c97-be51-45d0-818f-29c4d2e69318" containerID="21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650" exitCode=0 Nov 02 20:04:53.338099 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.337974 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" event={"ID":"69514c97-be51-45d0-818f-29c4d2e69318","Type":"ContainerDied","Data":"21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650"} Nov 02 20:04:53.338099 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.338037 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" Nov 02 20:04:53.338099 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.338072 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g" event={"ID":"69514c97-be51-45d0-818f-29c4d2e69318","Type":"ContainerDied","Data":"15b118ffd20bd3afc334a2630645953e4b071bbd1ef5f9db79275049b4305d69"} Nov 02 20:04:53.338099 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.338087 5850 scope.go:122] "RemoveContainer" containerID="21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650" Nov 02 20:04:53.348532 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.348515 5850 scope.go:122] "RemoveContainer" containerID="4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d" Nov 02 20:04:53.357674 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.357654 5850 scope.go:122] "RemoveContainer" containerID="21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650" Nov 02 20:04:53.357966 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:04:53.357943 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650\": container with ID starting with 21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650 not found: ID does not exist" containerID="21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650" Nov 02 20:04:53.358040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.357979 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650"} err="failed to get container status \"21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650\": rpc error: code = NotFound desc = could not find container \"21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650\": container with ID starting with 21bf9ac4e5bdf24e643827116d4caf305f9e7c1866c58819fb188b1c769d6650 not found: ID does not exist" Nov 02 20:04:53.358040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.358005 5850 scope.go:122] "RemoveContainer" containerID="4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d" Nov 02 20:04:53.358299 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:04:53.358280 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d\": container with ID starting with 4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d not found: ID does not exist" containerID="4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d" Nov 02 20:04:53.358341 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.358305 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d"} err="failed to get container status \"4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d\": rpc error: code = NotFound desc = could not find container \"4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d\": container with ID starting with 4f33dcdb4e26ab4d5efb620d0d1a74532a429e4abbf3dea98ea46aeaf127763d not found: ID does not exist" Nov 02 20:04:53.507263 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.507220 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g"] Nov 02 20:04:53.512321 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.512288 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0b7eed-openshift-default-6876f5f4dc-khf7g"] Nov 02 20:04:53.817837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:53.817803 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69514c97-be51-45d0-818f-29c4d2e69318" path="/var/lib/kubelet/pods/69514c97-be51-45d0-818f-29c4d2e69318/volumes" Nov 02 20:04:55.023311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.023268 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm"] Nov 02 20:04:55.024116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024092 5850 cpu_manager.go:414] "containerMap: removing container" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" Nov 02 20:04:55.024234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024117 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" Nov 02 20:04:55.024234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024150 5850 cpu_manager.go:414] "containerMap: removing container" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" Nov 02 20:04:55.024234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024159 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" Nov 02 20:04:55.024234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024177 5850 cpu_manager.go:414] "containerMap: removing container" podUID="c204339b-7ab7-43b1-afff-456ac421b761" containerName="maas-api" Nov 02 20:04:55.024234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024185 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c204339b-7ab7-43b1-afff-456ac421b761" containerName="maas-api" Nov 02 20:04:55.024234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024205 5850 cpu_manager.go:414] "containerMap: removing container" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" Nov 02 20:04:55.024234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024212 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" Nov 02 20:04:55.024234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024226 5850 cpu_manager.go:414] "containerMap: removing container" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" Nov 02 20:04:55.024234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024234 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" Nov 02 20:04:55.025316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024283 5850 cpu_manager.go:414] "containerMap: removing container" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" Nov 02 20:04:55.025316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024291 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" Nov 02 20:04:55.025316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024437 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" Nov 02 20:04:55.025316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024454 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" Nov 02 20:04:55.025316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024466 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" Nov 02 20:04:55.025316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024497 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a00dd58-66dc-4d13-97b8-acfcf52513e9" containerName="istio-proxy" Nov 02 20:04:55.025316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024507 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="69514c97-be51-45d0-818f-29c4d2e69318" containerName="istio-proxy" Nov 02 20:04:55.025316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024517 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="c204339b-7ab7-43b1-afff-456ac421b761" containerName="maas-api" Nov 02 20:04:55.025316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.024528 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="05973ae4-c92f-460b-a550-76b094ea96fd" containerName="istio-proxy" Nov 02 20:04:55.028686 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.028664 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.060766 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.060720 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfbwp\" (UniqueName: \"kubernetes.io/projected/265c981a-ccad-4e88-aeec-16b5efbce6ad-kube-api-access-tfbwp\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.060951 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.060775 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-metrics-certs\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.060990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.060943 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265c981a-ccad-4e88-aeec-16b5efbce6ad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.061083 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.061065 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.085132 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.085096 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm"] Nov 02 20:04:55.132829 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.132792 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-838bc1-serving-cert\"" Nov 02 20:04:55.162265 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.162224 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265c981a-ccad-4e88-aeec-16b5efbce6ad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.162449 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.162342 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.162552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.162463 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfbwp\" (UniqueName: \"kubernetes.io/projected/265c981a-ccad-4e88-aeec-16b5efbce6ad-kube-api-access-tfbwp\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.162552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.162540 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-metrics-certs\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.162643 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:04:55.162501 5850 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-serving-cert: secret "maas-api-e2e-inf-a-838bc1-serving-cert" not found Nov 02 20:04:55.162706 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:04:55.162691 5850 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls podName:265c981a-ccad-4e88-aeec-16b5efbce6ad nodeName:}" failed. No retries permitted until 2026-11-02 20:04:55.662666317 +0000 UTC m=+2154.005055376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls") pod "maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" (UID: "265c981a-ccad-4e88-aeec-16b5efbce6ad") : secret "maas-api-e2e-inf-a-838bc1-serving-cert" not found Nov 02 20:04:55.163116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.163095 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265c981a-ccad-4e88-aeec-16b5efbce6ad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.165076 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.165055 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-metrics-certs\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.247446 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.247414 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfbwp\" (UniqueName: \"kubernetes.io/projected/265c981a-ccad-4e88-aeec-16b5efbce6ad-kube-api-access-tfbwp\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.286357 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.286264 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:55.454689 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.454657 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:55.606305 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.606222 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:55.670622 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.670584 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.673116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.673086 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls\") pod \"maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:55.942811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:55.942773 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:56.317088 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:04:56.317051 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265c981a_ccad_4e88_aeec_16b5efbce6ad.slice/crio-c8353ff4b147346073f28308d3d101f439c71adb3ee1f7a5fb99f2faa5fb4f4a WatchSource:0}: Error finding container c8353ff4b147346073f28308d3d101f439c71adb3ee1f7a5fb99f2faa5fb4f4a: Status 404 returned error can't find the container with id c8353ff4b147346073f28308d3d101f439c71adb3ee1f7a5fb99f2faa5fb4f4a Nov 02 20:04:56.319851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:56.319828 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm"] Nov 02 20:04:56.367373 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:56.367335 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" event={"ID":"265c981a-ccad-4e88-aeec-16b5efbce6ad","Type":"ContainerStarted","Data":"c8353ff4b147346073f28308d3d101f439c71adb3ee1f7a5fb99f2faa5fb4f4a"} Nov 02 20:04:57.377077 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:57.377029 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" event={"ID":"265c981a-ccad-4e88-aeec-16b5efbce6ad","Type":"ContainerStarted","Data":"d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e"} Nov 02 20:04:57.377543 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:57.377251 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:04:57.496852 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:57.496793 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" podStartSLOduration=2.730539447 podStartE2EDuration="3.496777403s" totalImagesPullingTime="766.237956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:57.495776295 +0000 UTC m=+2155.838165376" watchObservedRunningTime="2026-11-02 20:04:57.496777403 +0000 UTC m=+2155.839166482" Nov 02 20:04:59.459758 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.459721 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457"] Nov 02 20:04:59.460177 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.459943 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" podUID="e621b751-a8ef-4a02-914e-d0c6135e1337" containerName="maas-api" containerID="cri-o://783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e" gracePeriod=30 Nov 02 20:04:59.606512 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.606443 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:59.801720 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.801691 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:04:59.835688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.835640 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e621b751-a8ef-4a02-914e-d0c6135e1337-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e621b751-a8ef-4a02-914e-d0c6135e1337-openshift-service-ca\") pod \"e621b751-a8ef-4a02-914e-d0c6135e1337\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " Nov 02 20:04:59.835920 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.835858 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls\") pod \"e621b751-a8ef-4a02-914e-d0c6135e1337\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " Nov 02 20:04:59.835994 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.835981 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e621b751-a8ef-4a02-914e-d0c6135e1337-kube-api-access-zvbw9\" (UniqueName: \"kubernetes.io/projected/e621b751-a8ef-4a02-914e-d0c6135e1337-kube-api-access-zvbw9\") pod \"e621b751-a8ef-4a02-914e-d0c6135e1337\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " Nov 02 20:04:59.836145 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.836115 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e621b751-a8ef-4a02-914e-d0c6135e1337-openshift-service-ca" pod "e621b751-a8ef-4a02-914e-d0c6135e1337" (UID: "e621b751-a8ef-4a02-914e-d0c6135e1337"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:59.836145 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.836138 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-metrics-certs\") pod \"e621b751-a8ef-4a02-914e-d0c6135e1337\" (UID: \"e621b751-a8ef-4a02-914e-d0c6135e1337\") " Nov 02 20:04:59.837218 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.837190 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e621b751-a8ef-4a02-914e-d0c6135e1337-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:59.841127 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.841096 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls" pod "e621b751-a8ef-4a02-914e-d0c6135e1337" (UID: "e621b751-a8ef-4a02-914e-d0c6135e1337"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:59.842384 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.842361 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-metrics-certs" pod "e621b751-a8ef-4a02-914e-d0c6135e1337" (UID: "e621b751-a8ef-4a02-914e-d0c6135e1337"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:59.842768 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.842753 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e621b751-a8ef-4a02-914e-d0c6135e1337-kube-api-access-zvbw9" pod "e621b751-a8ef-4a02-914e-d0c6135e1337" (UID: "e621b751-a8ef-4a02-914e-d0c6135e1337"). InnerVolumeSpecName "kube-api-access-zvbw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:59.939132 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.939095 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:59.939132 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.939129 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvbw9\" (UniqueName: \"kubernetes.io/projected/e621b751-a8ef-4a02-914e-d0c6135e1337-kube-api-access-zvbw9\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:59.939132 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:04:59.939141 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e621b751-a8ef-4a02-914e-d0c6135e1337-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:00.440774 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:00.440732 5850 generic.go:356] "Generic (PLEG): container finished" podID="e621b751-a8ef-4a02-914e-d0c6135e1337" containerID="783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e" exitCode=0 Nov 02 20:05:00.441161 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:00.441076 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" Nov 02 20:05:00.452273 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:00.441097 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" event={"ID":"e621b751-a8ef-4a02-914e-d0c6135e1337","Type":"ContainerDied","Data":"783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e"} Nov 02 20:05:00.452273 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:00.448643 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457" event={"ID":"e621b751-a8ef-4a02-914e-d0c6135e1337","Type":"ContainerDied","Data":"2ca043b0fde696b74621583bd8c035114a6acb08a86b031a29db99aba71950c3"} Nov 02 20:05:00.452273 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:00.448690 5850 scope.go:122] "RemoveContainer" containerID="783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e" Nov 02 20:05:00.542765 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:00.542730 5850 scope.go:122] "RemoveContainer" containerID="783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e" Nov 02 20:05:00.543212 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:00.543186 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e\": container with ID starting with 783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e not found: ID does not exist" containerID="783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e" Nov 02 20:05:00.543271 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:00.543230 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e"} err="failed to get container status \"783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e\": rpc error: code = NotFound desc = could not find container \"783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e\": container with ID starting with 783d3ba92ece64a95899541c8531d22ff3ed80b678134987ce5db0e9b033bb2e not found: ID does not exist" Nov 02 20:05:00.646020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:00.645981 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457"] Nov 02 20:05:00.697714 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:00.697628 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ffdc4d-6479544875-x6457"] Nov 02 20:05:01.125579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:01.125547 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:01.507261 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:01.507156 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:01.725331 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:01.725239 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:01.817310 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:01.817227 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e621b751-a8ef-4a02-914e-d0c6135e1337" path="/var/lib/kubelet/pods/e621b751-a8ef-4a02-914e-d0c6135e1337/volumes" Nov 02 20:05:02.063849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:02.063805 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:03.391160 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:03.391132 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:05:05.589087 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:05.589042 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:05.735138 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:05.735104 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:06.543486 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:06.543432 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:07.871810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:07.871764 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:08.027388 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:08.027350 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:08.211040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:08.210943 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:09.001879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.001844 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:09.516870 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.516836 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7"] Nov 02 20:05:09.517538 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.517519 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e621b751-a8ef-4a02-914e-d0c6135e1337" containerName="maas-api" Nov 02 20:05:09.517600 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.517539 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e621b751-a8ef-4a02-914e-d0c6135e1337" containerName="maas-api" Nov 02 20:05:09.517600 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.517569 5850 cpu_manager.go:414] "containerMap: removing container" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" Nov 02 20:05:09.517600 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.517578 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" Nov 02 20:05:09.517738 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.517716 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e621b751-a8ef-4a02-914e-d0c6135e1337" containerName="maas-api" Nov 02 20:05:09.517738 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.517734 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa77d63a-7ab5-4e82-9144-15039098659b" containerName="istio-proxy" Nov 02 20:05:09.521811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.521783 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.623498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.584705 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1db0e4-openshift-default-dockercfg-kfm8g\"" Nov 02 20:05:09.744429 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.744380 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7"] Nov 02 20:05:09.772693 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.772592 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istiod-ca-cert\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.772871 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.772709 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-podinfo\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.772871 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.772825 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-envoy\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.772871 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.772849 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25xc\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-kube-api-access-b25xc\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.773027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.772919 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-data\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.778079 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.778036 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-certs\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.778248 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.778102 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-token\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.778248 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.778181 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-credential-socket\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.778248 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.778207 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-socket\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.781681 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.781219 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:09.880262 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880216 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-certs\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880262 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880267 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-token\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880540 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880311 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-credential-socket\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880540 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880329 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-socket\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880667 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880557 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istiod-ca-cert\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880724 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880700 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-podinfo\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880776 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880719 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-certs\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880876 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880851 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-envoy\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880938 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880886 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25xc\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-kube-api-access-b25xc\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880938 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880895 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-socket\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.880938 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.880853 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-credential-socket\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.881119 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.881094 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-data\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.881385 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.881364 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-data\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.881542 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.881519 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istiod-ca-cert\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.883102 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.883079 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-envoy\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.883326 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.883308 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-podinfo\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.938922 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.938879 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-token\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.939108 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.939043 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25xc\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-kube-api-access-b25xc\") pod \"e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:09.944941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:09.944911 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:10.147560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:10.147446 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:10.381731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:10.381700 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7"] Nov 02 20:05:10.382597 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:10.382564 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:05:10.382743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:10.382647 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:05:10.382743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:10.382676 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:05:10.640615 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:10.640566 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" event={"ID":"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0","Type":"ContainerStarted","Data":"256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317"} Nov 02 20:05:10.640832 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:10.640665 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" event={"ID":"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0","Type":"ContainerStarted","Data":"a3c4691486f7b7533f630c2eb5d8a4bbc9124c915a0669479480af8a96c3e277"} Nov 02 20:05:10.763359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:10.763249 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" podStartSLOduration=1.7632302800000001 podStartE2EDuration="1.76323028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:10.762331924 +0000 UTC m=+2169.104721025" watchObservedRunningTime="2026-11-02 20:05:10.76323028 +0000 UTC m=+2169.105619362" Nov 02 20:05:10.946574 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:10.946541 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:11.324922 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:11.324891 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:11.950981 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:11.950947 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:12.653228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:12.653196 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:12.654254 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:12.654232 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:05:13.520246 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:13.520184 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:13.691269 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:13.691223 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:13.860840 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:13.860726 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:14.762955 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:14.762917 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:05:15.215965 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.215927 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb"] Nov 02 20:05:15.216170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.216127 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" podUID="4408de12-184d-4185-9f4e-b5eda75b116a" containerName="maas-api" containerID="cri-o://62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b" gracePeriod=30 Nov 02 20:05:15.335996 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.335962 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:15.514749 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.514667 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:15.552106 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.552080 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 20:05:15.686305 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.686266 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4408de12-184d-4185-9f4e-b5eda75b116a-kube-api-access-2bjdw\" (UniqueName: \"kubernetes.io/projected/4408de12-184d-4185-9f4e-b5eda75b116a-kube-api-access-2bjdw\") pod \"4408de12-184d-4185-9f4e-b5eda75b116a\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " Nov 02 20:05:15.687006 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.686980 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-metrics-certs\") pod \"4408de12-184d-4185-9f4e-b5eda75b116a\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " Nov 02 20:05:15.687245 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.687229 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls\") pod \"4408de12-184d-4185-9f4e-b5eda75b116a\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " Nov 02 20:05:15.687414 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.687395 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4408de12-184d-4185-9f4e-b5eda75b116a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4408de12-184d-4185-9f4e-b5eda75b116a-openshift-service-ca\") pod \"4408de12-184d-4185-9f4e-b5eda75b116a\" (UID: \"4408de12-184d-4185-9f4e-b5eda75b116a\") " Nov 02 20:05:15.690031 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.690001 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4408de12-184d-4185-9f4e-b5eda75b116a-openshift-service-ca" pod "4408de12-184d-4185-9f4e-b5eda75b116a" (UID: "4408de12-184d-4185-9f4e-b5eda75b116a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:15.740910 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.732059 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-metrics-certs" pod "4408de12-184d-4185-9f4e-b5eda75b116a" (UID: "4408de12-184d-4185-9f4e-b5eda75b116a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:15.751795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.743744 5850 generic.go:356] "Generic (PLEG): container finished" podID="4408de12-184d-4185-9f4e-b5eda75b116a" containerID="62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b" exitCode=0 Nov 02 20:05:15.751795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.743894 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" event={"ID":"4408de12-184d-4185-9f4e-b5eda75b116a","Type":"ContainerDied","Data":"62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b"} Nov 02 20:05:15.751795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.743981 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" event={"ID":"4408de12-184d-4185-9f4e-b5eda75b116a","Type":"ContainerDied","Data":"2318698ff3f090868562a054daa8a6468c9354c85216c70c943caa402307b84e"} Nov 02 20:05:15.751795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.744020 5850 scope.go:122] "RemoveContainer" containerID="62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b" Nov 02 20:05:15.751795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.744189 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb" Nov 02 20:05:15.751795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.750378 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4408de12-184d-4185-9f4e-b5eda75b116a-kube-api-access-2bjdw" pod "4408de12-184d-4185-9f4e-b5eda75b116a" (UID: "4408de12-184d-4185-9f4e-b5eda75b116a"). InnerVolumeSpecName "kube-api-access-2bjdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:15.751795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.750870 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls" pod "4408de12-184d-4185-9f4e-b5eda75b116a" (UID: "4408de12-184d-4185-9f4e-b5eda75b116a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:15.790417 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.790323 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bjdw\" (UniqueName: \"kubernetes.io/projected/4408de12-184d-4185-9f4e-b5eda75b116a-kube-api-access-2bjdw\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:15.790417 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.790366 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:15.790417 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.790377 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4408de12-184d-4185-9f4e-b5eda75b116a-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:15.790417 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.790386 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4408de12-184d-4185-9f4e-b5eda75b116a-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:15.795163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.795132 5850 scope.go:122] "RemoveContainer" containerID="62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b" Nov 02 20:05:15.795656 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:15.795638 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b\": container with ID starting with 62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b not found: ID does not exist" containerID="62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b" Nov 02 20:05:15.795795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:15.795772 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b"} err="failed to get container status \"62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b\": rpc error: code = NotFound desc = could not find container \"62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b\": container with ID starting with 62fc63645133b9159750e964878650535031665b9e421122d330d5699c53b65b not found: ID does not exist" Nov 02 20:05:16.200140 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.200099 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb"] Nov 02 20:05:16.239729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.239678 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-32abda-7464db4d45-b4mzb"] Nov 02 20:05:16.298300 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.298259 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv"] Nov 02 20:05:16.299020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.298985 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" podUID="4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" containerName="maas-api" containerID="cri-o://40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3" gracePeriod=30 Nov 02 20:05:16.419054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.419013 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:16.631146 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.631113 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:05:16.722241 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.722189 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-kube-api-access-q47x8\" (UniqueName: \"kubernetes.io/projected/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-kube-api-access-q47x8\") pod \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " Nov 02 20:05:16.722523 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.722382 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-maas-api-tls\") pod \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " Nov 02 20:05:16.722627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.722536 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-metrics-certs\") pod \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " Nov 02 20:05:16.722627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.722585 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-openshift-service-ca\") pod \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\" (UID: \"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b\") " Nov 02 20:05:16.727850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.727572 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-maas-api-tls" pod "4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" (UID: "4502b57c-0a04-448b-b2e5-90f9e9dd6e1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:16.730880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.730824 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-kube-api-access-q47x8" pod "4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" (UID: "4502b57c-0a04-448b-b2e5-90f9e9dd6e1b"). InnerVolumeSpecName "kube-api-access-q47x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:16.733159 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.733114 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-metrics-certs" pod "4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" (UID: "4502b57c-0a04-448b-b2e5-90f9e9dd6e1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:16.737003 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.736941 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-openshift-service-ca" pod "4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" (UID: "4502b57c-0a04-448b-b2e5-90f9e9dd6e1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:16.792538 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.792466 5850 generic.go:356] "Generic (PLEG): container finished" podID="4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" containerID="40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3" exitCode=0 Nov 02 20:05:16.792538 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.792503 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" event={"ID":"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b","Type":"ContainerDied","Data":"40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3"} Nov 02 20:05:16.793048 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.792538 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" Nov 02 20:05:16.793048 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.792568 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv" event={"ID":"4502b57c-0a04-448b-b2e5-90f9e9dd6e1b","Type":"ContainerDied","Data":"9e42f81d9ec21fda54772ab9323465f9e66bfec71feac9a2cb6f0756031d9bbc"} Nov 02 20:05:16.793048 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.792589 5850 scope.go:122] "RemoveContainer" containerID="40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3" Nov 02 20:05:16.802936 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.802820 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:16.804988 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.804959 5850 scope.go:122] "RemoveContainer" containerID="40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3" Nov 02 20:05:16.805319 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:16.805298 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3\": container with ID starting with 40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3 not found: ID does not exist" containerID="40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3" Nov 02 20:05:16.805428 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.805329 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3"} err="failed to get container status \"40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3\": rpc error: code = NotFound desc = could not find container \"40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3\": container with ID starting with 40ecca7dbc3918d5c824db0e856a40926774895b087363ff693e01fbf189d6e3 not found: ID does not exist" Nov 02 20:05:16.824975 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.824939 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:16.824975 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.824972 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:16.825190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.824982 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:16.825190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:16.824993 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q47x8\" (UniqueName: \"kubernetes.io/projected/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b-kube-api-access-q47x8\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:17.018332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.018227 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:17.023466 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.023355 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv"] Nov 02 20:05:17.036895 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.036851 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b0a12-d64f7b797-xxrxv"] Nov 02 20:05:17.300463 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.300369 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:17.330530 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.330467 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4"] Nov 02 20:05:17.331140 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.331127 5850 cpu_manager.go:414] "containerMap: removing container" podUID="4408de12-184d-4185-9f4e-b5eda75b116a" containerName="maas-api" Nov 02 20:05:17.331195 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.331141 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4408de12-184d-4185-9f4e-b5eda75b116a" containerName="maas-api" Nov 02 20:05:17.331195 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.331178 5850 cpu_manager.go:414] "containerMap: removing container" podUID="4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" containerName="maas-api" Nov 02 20:05:17.331195 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.331184 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" containerName="maas-api" Nov 02 20:05:17.331295 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.331269 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" containerName="maas-api" Nov 02 20:05:17.331295 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.331284 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="4408de12-184d-4185-9f4e-b5eda75b116a" containerName="maas-api" Nov 02 20:05:17.335006 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.334988 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:17.389589 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.389546 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1db0e4\"" Nov 02 20:05:17.389589 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.389586 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1db0e4-dockercfg-rvcf2\"" Nov 02 20:05:17.391528 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.391496 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4"] Nov 02 20:05:17.499499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.491353 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4\" (UID: \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\") " pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:17.542544 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.516810 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:17.542544 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.522992 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-kube-api-access-vv6qw\") pod \"payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4\" (UID: \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\") " pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:17.559785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.559683 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn"] Nov 02 20:05:17.595822 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.595782 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:17.609252 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.607755 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn"] Nov 02 20:05:17.624336 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.624303 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-kube-api-access-vv6qw\") pod \"payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4\" (UID: \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\") " pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:17.624567 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.624394 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4\" (UID: \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\") " pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:17.625390 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.625366 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4\" (UID: \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\") " pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:17.684046 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.684008 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-kube-api-access-vv6qw\") pod \"payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4\" (UID: \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\") " pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:17.726605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.726562 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/1b6821f8-53b3-4ec3-b983-66e93eda9e86-kube-api-access-f5lxl\") pod \"payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn\" (UID: \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:17.726826 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.726768 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6821f8-53b3-4ec3-b983-66e93eda9e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn\" (UID: \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:17.815797 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.815717 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4408de12-184d-4185-9f4e-b5eda75b116a" path="/var/lib/kubelet/pods/4408de12-184d-4185-9f4e-b5eda75b116a/volumes" Nov 02 20:05:17.816200 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.816186 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4502b57c-0a04-448b-b2e5-90f9e9dd6e1b" path="/var/lib/kubelet/pods/4502b57c-0a04-448b-b2e5-90f9e9dd6e1b/volumes" Nov 02 20:05:17.827901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.827868 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/1b6821f8-53b3-4ec3-b983-66e93eda9e86-kube-api-access-f5lxl\") pod \"payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn\" (UID: \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:17.828047 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.827984 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6821f8-53b3-4ec3-b983-66e93eda9e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn\" (UID: \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:17.828754 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.828733 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6821f8-53b3-4ec3-b983-66e93eda9e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn\" (UID: \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:17.886676 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.886647 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/1b6821f8-53b3-4ec3-b983-66e93eda9e86-kube-api-access-f5lxl\") pod \"payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn\" (UID: \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:17.925730 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.925681 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:17.947954 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:17.947902 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:18.017829 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.017795 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:18.173213 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.173179 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:18.323506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.309240 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn"] Nov 02 20:05:18.329528 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.329191 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:18.359493 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.359166 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4"] Nov 02 20:05:18.386730 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:05:18.386684 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1c21e9_8348_4d6d_bff8_fda75236b9e3.slice/crio-3843c4250292d75fb193cca99dc2a11c2c846685c2ae4be3a40a095fb94b50d3 WatchSource:0}: Error finding container 3843c4250292d75fb193cca99dc2a11c2c846685c2ae4be3a40a095fb94b50d3: Status 404 returned error can't find the container with id 3843c4250292d75fb193cca99dc2a11c2c846685c2ae4be3a40a095fb94b50d3 Nov 02 20:05:18.523291 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.523259 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:18.808504 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.808394 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" event={"ID":"1b6821f8-53b3-4ec3-b983-66e93eda9e86","Type":"ContainerStarted","Data":"22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168"} Nov 02 20:05:18.808504 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.808450 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" event={"ID":"1b6821f8-53b3-4ec3-b983-66e93eda9e86","Type":"ContainerStarted","Data":"507039f289bf306fea013bd68ebe1ac9b33dce9b9bc1e8b4515f17bebe2ea97d"} Nov 02 20:05:18.808504 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.808503 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:18.810187 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.810162 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" event={"ID":"5c1c21e9-8348-4d6d-bff8-fda75236b9e3","Type":"ContainerStarted","Data":"57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662"} Nov 02 20:05:18.810308 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.810208 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" event={"ID":"5c1c21e9-8348-4d6d-bff8-fda75236b9e3","Type":"ContainerStarted","Data":"3843c4250292d75fb193cca99dc2a11c2c846685c2ae4be3a40a095fb94b50d3"} Nov 02 20:05:18.810370 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.810315 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:18.929219 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.929165 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" podStartSLOduration=1.929149258 podStartE2EDuration="1.929149258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:18.926347544 +0000 UTC m=+2177.268736625" watchObservedRunningTime="2026-11-02 20:05:18.929149258 +0000 UTC m=+2177.271538338" Nov 02 20:05:18.997673 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:18.997643 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:19.057239 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:19.051895 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" podStartSLOduration=2.051875453 podStartE2EDuration="2.051875453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:19.051221893 +0000 UTC m=+2177.393610974" watchObservedRunningTime="2026-11-02 20:05:19.051875453 +0000 UTC m=+2177.394264534" Nov 02 20:05:23.087790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:23.087755 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:23.710313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:23.710278 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:23.902505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:23.886534 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:24.176439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:24.176402 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:24.216596 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:24.216554 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j"] Nov 02 20:05:24.218092 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:24.217830 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" containerID="cri-o://5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7" gracePeriod=30 Nov 02 20:05:24.536646 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:24.536549 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:25.241965 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:25.241928 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:25.410137 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:25.410104 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:25.575395 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:25.575310 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:26.122724 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:26.122682 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:28.073750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:28.073718 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:29.478697 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.478654 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94"] Nov 02 20:05:29.529760 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.529723 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5"] Nov 02 20:05:29.619124 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.619099 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 20:05:29.661715 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.661681 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:29.737270 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737231 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istiod-ca-cert\") pod \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " Nov 02 20:05:29.737270 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737278 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-data\") pod \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " Nov 02 20:05:29.737518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737370 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-kube-api-access-gzdbf\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-kube-api-access-gzdbf\") pod \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " Nov 02 20:05:29.737518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737418 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-socket\") pod \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " Nov 02 20:05:29.737610 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737526 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-podinfo\") pod \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " Nov 02 20:05:29.737610 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737568 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-envoy\") pod \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " Nov 02 20:05:29.737718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737624 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-token\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-token\") pod \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " Nov 02 20:05:29.737718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737668 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-credential-socket\") pod \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " Nov 02 20:05:29.737718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737691 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-data" pod "cec5c949-3bd9-4ff6-a7be-76c6bf064239" (UID: "cec5c949-3bd9-4ff6-a7be-76c6bf064239"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:29.737898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737773 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-certs\") pod \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\" (UID: \"cec5c949-3bd9-4ff6-a7be-76c6bf064239\") " Nov 02 20:05:29.737898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737759 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istiod-ca-cert" pod "cec5c949-3bd9-4ff6-a7be-76c6bf064239" (UID: "cec5c949-3bd9-4ff6-a7be-76c6bf064239"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:29.737898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.737840 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-socket" pod "cec5c949-3bd9-4ff6-a7be-76c6bf064239" (UID: "cec5c949-3bd9-4ff6-a7be-76c6bf064239"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:29.738060 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.738043 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-credential-socket" pod "cec5c949-3bd9-4ff6-a7be-76c6bf064239" (UID: "cec5c949-3bd9-4ff6-a7be-76c6bf064239"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:29.738176 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.738146 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-certs" pod "cec5c949-3bd9-4ff6-a7be-76c6bf064239" (UID: "cec5c949-3bd9-4ff6-a7be-76c6bf064239"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:29.738556 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.738532 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.738657 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.738562 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.738657 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.738577 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.738657 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.738592 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.738657 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.738607 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.739966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.739936 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-envoy" pod "cec5c949-3bd9-4ff6-a7be-76c6bf064239" (UID: "cec5c949-3bd9-4ff6-a7be-76c6bf064239"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:29.740082 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.739980 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-podinfo" pod "cec5c949-3bd9-4ff6-a7be-76c6bf064239" (UID: "cec5c949-3bd9-4ff6-a7be-76c6bf064239"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:05:29.740082 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.740023 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-kube-api-access-gzdbf" pod "cec5c949-3bd9-4ff6-a7be-76c6bf064239" (UID: "cec5c949-3bd9-4ff6-a7be-76c6bf064239"). InnerVolumeSpecName "kube-api-access-gzdbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:29.740082 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.740058 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-token" pod "cec5c949-3bd9-4ff6-a7be-76c6bf064239" (UID: "cec5c949-3bd9-4ff6-a7be-76c6bf064239"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:29.820660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.820626 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:05:29.820825 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.820701 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:05:29.839747 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.839711 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzdbf\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-kube-api-access-gzdbf\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.839747 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.839746 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.839948 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.839759 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.839948 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.839771 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec5c949-3bd9-4ff6-a7be-76c6bf064239-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.932586 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.932495 5850 generic.go:356] "Generic (PLEG): container finished" podID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerID="5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7" exitCode=0 Nov 02 20:05:29.932586 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.932528 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" event={"ID":"cec5c949-3bd9-4ff6-a7be-76c6bf064239","Type":"ContainerDied","Data":"5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7"} Nov 02 20:05:29.932586 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.932570 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" Nov 02 20:05:29.932866 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.932614 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j" event={"ID":"cec5c949-3bd9-4ff6-a7be-76c6bf064239","Type":"ContainerDied","Data":"6545dc1ba087d2319de4662df6469117d133bc356499912410f73a6ec13934b5"} Nov 02 20:05:29.932866 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.932640 5850 scope.go:122] "RemoveContainer" containerID="5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7" Nov 02 20:05:29.943269 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.943247 5850 scope.go:122] "RemoveContainer" containerID="6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630" Nov 02 20:05:29.954638 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.954615 5850 scope.go:122] "RemoveContainer" containerID="5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7" Nov 02 20:05:29.954956 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:29.954934 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7\": container with ID starting with 5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7 not found: ID does not exist" containerID="5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7" Nov 02 20:05:29.955011 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.954966 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7"} err="failed to get container status \"5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7\": rpc error: code = NotFound desc = could not find container \"5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7\": container with ID starting with 5d0c3fe6daf4b4a11fe9e5250cb40136d9867694057abaf623609e35f328bcd7 not found: ID does not exist" Nov 02 20:05:29.955011 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.954986 5850 scope.go:122] "RemoveContainer" containerID="6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630" Nov 02 20:05:29.955224 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:29.955207 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630\": container with ID starting with 6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630 not found: ID does not exist" containerID="6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630" Nov 02 20:05:29.955283 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:29.955228 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630"} err="failed to get container status \"6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630\": rpc error: code = NotFound desc = could not find container \"6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630\": container with ID starting with 6bbf4ad73dadbbdf453e77a5a304d1435ea787f75bba98129322364109d72630 not found: ID does not exist" Nov 02 20:05:30.184999 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:30.184882 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j"] Nov 02 20:05:30.187707 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:30.187679 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32abda-openshift-default-66b9d6d677-2r58j"] Nov 02 20:05:31.062102 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:31.062062 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8"] Nov 02 20:05:31.071431 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:31.071384 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm"] Nov 02 20:05:31.241178 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:31.241146 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:31.816775 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:31.816742 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" path="/var/lib/kubelet/pods/cec5c949-3bd9-4ff6-a7be-76c6bf064239/volumes" Nov 02 20:05:32.734790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:32.734755 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:37.075044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:37.074999 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:37.231267 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:37.231237 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:37.384935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:37.384898 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:39.384005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:39.383962 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg"] Nov 02 20:05:39.666359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:39.666264 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:39.822303 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:39.822268 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:41.770170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:41.770136 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:44.384769 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.384721 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" podUID="dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" containerName="payload-pre-processing" containerID="cri-o://fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed" gracePeriod=25 Nov 02 20:05:44.501214 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.501172 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" containerID="cri-o://3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a" gracePeriod=45 Nov 02 20:05:44.553701 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.553631 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" containerID="cri-o://e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548" gracePeriod=45 Nov 02 20:05:44.781193 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:44.781146 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:05:44.852413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.852384 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 20:05:44.855659 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.855637 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 20:05:44.899315 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899281 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-dshm\") pod \"d90f1e76-4c42-4056-85f2-b3502e2e4359\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " Nov 02 20:05:44.899541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899325 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d90f1e76-4c42-4056-85f2-b3502e2e4359-tls-certs\" (UniqueName: \"kubernetes.io/secret/d90f1e76-4c42-4056-85f2-b3502e2e4359-tls-certs\") pod \"d90f1e76-4c42-4056-85f2-b3502e2e4359\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " Nov 02 20:05:44.899541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899379 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-home\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-home\") pod \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " Nov 02 20:05:44.899541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899418 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90f1e76-4c42-4056-85f2-b3502e2e4359-kube-api-access-fk4ch\" (UniqueName: \"kubernetes.io/projected/d90f1e76-4c42-4056-85f2-b3502e2e4359-kube-api-access-fk4ch\") pod \"d90f1e76-4c42-4056-85f2-b3502e2e4359\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " Nov 02 20:05:44.899541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899448 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-model-cache\") pod \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " Nov 02 20:05:44.899541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899507 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-dshm\") pod \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " Nov 02 20:05:44.899846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899593 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-tmp-dir\") pod \"d90f1e76-4c42-4056-85f2-b3502e2e4359\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " Nov 02 20:05:44.899846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899682 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-home\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-home\") pod \"d90f1e76-4c42-4056-85f2-b3502e2e4359\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " Nov 02 20:05:44.899846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899744 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2079ca4-fcae-4d5e-8091-78fddd27fe84-kube-api-access-mqnmn\" (UniqueName: \"kubernetes.io/projected/f2079ca4-fcae-4d5e-8091-78fddd27fe84-kube-api-access-mqnmn\") pod \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " Nov 02 20:05:44.899846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899760 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-model-cache" pod "f2079ca4-fcae-4d5e-8091-78fddd27fe84" (UID: "f2079ca4-fcae-4d5e-8091-78fddd27fe84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:44.900059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899890 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tls-certs\") pod \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " Nov 02 20:05:44.900059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899935 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-model-cache\") pod \"d90f1e76-4c42-4056-85f2-b3502e2e4359\" (UID: \"d90f1e76-4c42-4056-85f2-b3502e2e4359\") " Nov 02 20:05:44.900059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.899970 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-home" pod "f2079ca4-fcae-4d5e-8091-78fddd27fe84" (UID: "f2079ca4-fcae-4d5e-8091-78fddd27fe84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:44.900059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900002 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-tmp-dir" pod "d90f1e76-4c42-4056-85f2-b3502e2e4359" (UID: "d90f1e76-4c42-4056-85f2-b3502e2e4359"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:44.900059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900009 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tmp-dir\") pod \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\" (UID: \"f2079ca4-fcae-4d5e-8091-78fddd27fe84\") " Nov 02 20:05:44.900059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900040 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-home" pod "d90f1e76-4c42-4056-85f2-b3502e2e4359" (UID: "d90f1e76-4c42-4056-85f2-b3502e2e4359"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:44.900360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900228 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tmp-dir" pod "f2079ca4-fcae-4d5e-8091-78fddd27fe84" (UID: "f2079ca4-fcae-4d5e-8091-78fddd27fe84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:44.900360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900235 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-model-cache" pod "d90f1e76-4c42-4056-85f2-b3502e2e4359" (UID: "d90f1e76-4c42-4056-85f2-b3502e2e4359"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:44.900703 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900667 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:44.900703 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900694 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:44.900887 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900709 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:44.900887 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900721 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:44.900887 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900733 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:44.900887 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.900745 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:44.902381 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.902362 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-dshm" pod "d90f1e76-4c42-4056-85f2-b3502e2e4359" (UID: "d90f1e76-4c42-4056-85f2-b3502e2e4359"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:44.902555 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.902531 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tls-certs" pod "f2079ca4-fcae-4d5e-8091-78fddd27fe84" (UID: "f2079ca4-fcae-4d5e-8091-78fddd27fe84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:44.902622 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.902541 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90f1e76-4c42-4056-85f2-b3502e2e4359-tls-certs" pod "d90f1e76-4c42-4056-85f2-b3502e2e4359" (UID: "d90f1e76-4c42-4056-85f2-b3502e2e4359"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:44.902622 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.902569 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90f1e76-4c42-4056-85f2-b3502e2e4359-kube-api-access-fk4ch" pod "d90f1e76-4c42-4056-85f2-b3502e2e4359" (UID: "d90f1e76-4c42-4056-85f2-b3502e2e4359"). InnerVolumeSpecName "kube-api-access-fk4ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:44.902707 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.902660 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-dshm" pod "f2079ca4-fcae-4d5e-8091-78fddd27fe84" (UID: "f2079ca4-fcae-4d5e-8091-78fddd27fe84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:44.902707 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:44.902689 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2079ca4-fcae-4d5e-8091-78fddd27fe84-kube-api-access-mqnmn" pod "f2079ca4-fcae-4d5e-8091-78fddd27fe84" (UID: "f2079ca4-fcae-4d5e-8091-78fddd27fe84"). InnerVolumeSpecName "kube-api-access-mqnmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:45.002039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.001956 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqnmn\" (UniqueName: \"kubernetes.io/projected/f2079ca4-fcae-4d5e-8091-78fddd27fe84-kube-api-access-mqnmn\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:45.002039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.001990 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2079ca4-fcae-4d5e-8091-78fddd27fe84-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:45.002039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.002000 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90f1e76-4c42-4056-85f2-b3502e2e4359-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:45.002039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.002009 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90f1e76-4c42-4056-85f2-b3502e2e4359-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:45.002039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.002018 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk4ch\" (UniqueName: \"kubernetes.io/projected/d90f1e76-4c42-4056-85f2-b3502e2e4359-kube-api-access-fk4ch\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:45.002039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.002028 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2079ca4-fcae-4d5e-8091-78fddd27fe84-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:45.068436 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.068401 5850 generic.go:356] "Generic (PLEG): container finished" podID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerID="3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a" exitCode=0 Nov 02 20:05:45.068669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.068461 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" Nov 02 20:05:45.068669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.068579 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" event={"ID":"f2079ca4-fcae-4d5e-8091-78fddd27fe84","Type":"ContainerDied","Data":"3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a"} Nov 02 20:05:45.068669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.068617 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94" event={"ID":"f2079ca4-fcae-4d5e-8091-78fddd27fe84","Type":"ContainerDied","Data":"19dc4e86aa2ffd3fdd8e51b40a3e8dec68f90d4fcf4bd409d5e7a6e680e4aeaa"} Nov 02 20:05:45.068669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.068637 5850 scope.go:122] "RemoveContainer" containerID="3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a" Nov 02 20:05:45.073447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.073421 5850 generic.go:356] "Generic (PLEG): container finished" podID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerID="e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548" exitCode=0 Nov 02 20:05:45.073649 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.073606 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" Nov 02 20:05:45.073839 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.073547 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" event={"ID":"d90f1e76-4c42-4056-85f2-b3502e2e4359","Type":"ContainerDied","Data":"e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548"} Nov 02 20:05:45.073839 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.073711 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5" event={"ID":"d90f1e76-4c42-4056-85f2-b3502e2e4359","Type":"ContainerDied","Data":"8fac629c4f286cdfc3ecc244eecd9f5518a080c7db2c543c1e82cd86d7a1c96d"} Nov 02 20:05:45.083964 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.083829 5850 scope.go:122] "RemoveContainer" containerID="3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a" Nov 02 20:05:45.085245 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:45.085217 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a\": container with ID starting with 3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a not found: ID does not exist" containerID="3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a" Nov 02 20:05:45.085358 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.085259 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a"} err="failed to get container status \"3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a\": rpc error: code = NotFound desc = could not find container \"3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a\": container with ID starting with 3922ed86b197f0603fd11a90c2bc9b0f9a451d96725b17123d9eb1988c77ef2a not found: ID does not exist" Nov 02 20:05:45.085358 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.085282 5850 scope.go:122] "RemoveContainer" containerID="e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548" Nov 02 20:05:45.099448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.098940 5850 scope.go:122] "RemoveContainer" containerID="e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548" Nov 02 20:05:45.099860 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:45.099813 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548\": container with ID starting with e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548 not found: ID does not exist" containerID="e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548" Nov 02 20:05:45.099973 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.099873 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548"} err="failed to get container status \"e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548\": rpc error: code = NotFound desc = could not find container \"e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548\": container with ID starting with e115bc65fae4b837fb4139d79661a2786fb92aabc94cc7809e377ba47fb83548 not found: ID does not exist" Nov 02 20:05:45.210684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.210645 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5"] Nov 02 20:05:45.213355 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.213327 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/simulator-w5-1d869e-kserve-6f6d56ff5f-9jzx5"] Nov 02 20:05:45.322387 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.322285 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94"] Nov 02 20:05:45.325675 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.325645 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1d869e/distinct-w5-1d869e-kserve-d7788465-6fw94"] Nov 02 20:05:45.816714 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.816678 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" path="/var/lib/kubelet/pods/d90f1e76-4c42-4056-85f2-b3502e2e4359/volumes" Nov 02 20:05:45.817169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.817155 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" path="/var/lib/kubelet/pods/f2079ca4-fcae-4d5e-8091-78fddd27fe84/volumes" Nov 02 20:05:45.832361 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:45.832323 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" podUID="dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:46.084542 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.084448 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" containerID="cri-o://0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f" gracePeriod=45 Nov 02 20:05:46.095945 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.095902 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" containerID="cri-o://5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2" gracePeriod=45 Nov 02 20:05:46.117013 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.116984 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:46.174053 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.174004 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused" Nov 02 20:05:46.400279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.400250 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 20:05:46.404051 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.404024 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 20:05:46.425996 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.425960 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-home\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-home\") pod \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " Nov 02 20:05:46.425996 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426000 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-dshm\") pod \"5664696d-245a-433b-8659-c44fd4a93e26\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " Nov 02 20:05:46.426228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426023 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-dshm\") pod \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " Nov 02 20:05:46.426228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426061 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-kube-api-access-g9jfv\" (UniqueName: \"kubernetes.io/projected/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-kube-api-access-g9jfv\") pod \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " Nov 02 20:05:46.426228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426091 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5664696d-245a-433b-8659-c44fd4a93e26-kube-api-access-h8xgq\" (UniqueName: \"kubernetes.io/projected/5664696d-245a-433b-8659-c44fd4a93e26-kube-api-access-h8xgq\") pod \"5664696d-245a-433b-8659-c44fd4a93e26\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " Nov 02 20:05:46.426228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426109 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-tmp-dir\") pod \"5664696d-245a-433b-8659-c44fd4a93e26\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " Nov 02 20:05:46.426228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426141 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tmp-dir\") pod \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " Nov 02 20:05:46.426228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426160 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-home\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-home\") pod \"5664696d-245a-433b-8659-c44fd4a93e26\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " Nov 02 20:05:46.426228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426221 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-model-cache\") pod \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " Nov 02 20:05:46.426654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426292 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-home" pod "f1fef2ff-3579-43ff-b49b-d86e0d0698e0" (UID: "f1fef2ff-3579-43ff-b49b-d86e0d0698e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.426654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426335 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5664696d-245a-433b-8659-c44fd4a93e26-tls-certs\" (UniqueName: \"kubernetes.io/secret/5664696d-245a-433b-8659-c44fd4a93e26-tls-certs\") pod \"5664696d-245a-433b-8659-c44fd4a93e26\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " Nov 02 20:05:46.426654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426374 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-model-cache\") pod \"5664696d-245a-433b-8659-c44fd4a93e26\" (UID: \"5664696d-245a-433b-8659-c44fd4a93e26\") " Nov 02 20:05:46.426654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426424 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-tmp-dir" pod "5664696d-245a-433b-8659-c44fd4a93e26" (UID: "5664696d-245a-433b-8659-c44fd4a93e26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.426654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426510 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tmp-dir" pod "f1fef2ff-3579-43ff-b49b-d86e0d0698e0" (UID: "f1fef2ff-3579-43ff-b49b-d86e0d0698e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.426654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426562 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-home" pod "5664696d-245a-433b-8659-c44fd4a93e26" (UID: "5664696d-245a-433b-8659-c44fd4a93e26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.426961 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426663 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tls-certs\") pod \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\" (UID: \"f1fef2ff-3579-43ff-b49b-d86e0d0698e0\") " Nov 02 20:05:46.426961 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426671 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-model-cache" pod "f1fef2ff-3579-43ff-b49b-d86e0d0698e0" (UID: "f1fef2ff-3579-43ff-b49b-d86e0d0698e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.426961 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.426759 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-model-cache" pod "5664696d-245a-433b-8659-c44fd4a93e26" (UID: "5664696d-245a-433b-8659-c44fd4a93e26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.427361 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.427339 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.427440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.427365 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.427440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.427379 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.427440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.427392 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.427440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.427406 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.427440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.427419 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.428839 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.428816 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tls-certs" pod "f1fef2ff-3579-43ff-b49b-d86e0d0698e0" (UID: "f1fef2ff-3579-43ff-b49b-d86e0d0698e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:46.428946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.428813 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5664696d-245a-433b-8659-c44fd4a93e26-kube-api-access-h8xgq" pod "5664696d-245a-433b-8659-c44fd4a93e26" (UID: "5664696d-245a-433b-8659-c44fd4a93e26"). InnerVolumeSpecName "kube-api-access-h8xgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:46.428946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.428879 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-dshm" pod "5664696d-245a-433b-8659-c44fd4a93e26" (UID: "5664696d-245a-433b-8659-c44fd4a93e26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.428946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.428897 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-kube-api-access-g9jfv" pod "f1fef2ff-3579-43ff-b49b-d86e0d0698e0" (UID: "f1fef2ff-3579-43ff-b49b-d86e0d0698e0"). InnerVolumeSpecName "kube-api-access-g9jfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:46.429075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.429059 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5664696d-245a-433b-8659-c44fd4a93e26-tls-certs" pod "5664696d-245a-433b-8659-c44fd4a93e26" (UID: "5664696d-245a-433b-8659-c44fd4a93e26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:46.429135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.429117 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-dshm" pod "f1fef2ff-3579-43ff-b49b-d86e0d0698e0" (UID: "f1fef2ff-3579-43ff-b49b-d86e0d0698e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.529439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.529397 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5664696d-245a-433b-8659-c44fd4a93e26-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.529439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.529442 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.529439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.529457 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5664696d-245a-433b-8659-c44fd4a93e26-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.529439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.529485 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.529439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.529500 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jfv\" (UniqueName: \"kubernetes.io/projected/f1fef2ff-3579-43ff-b49b-d86e0d0698e0-kube-api-access-g9jfv\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.529439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:46.529515 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8xgq\" (UniqueName: \"kubernetes.io/projected/5664696d-245a-433b-8659-c44fd4a93e26-kube-api-access-h8xgq\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:47.089020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.088981 5850 generic.go:356] "Generic (PLEG): container finished" podID="5664696d-245a-433b-8659-c44fd4a93e26" containerID="5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2" exitCode=0 Nov 02 20:05:47.089497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.089033 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" Nov 02 20:05:47.089497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.089062 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" event={"ID":"5664696d-245a-433b-8659-c44fd4a93e26","Type":"ContainerDied","Data":"5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2"} Nov 02 20:05:47.089497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.089112 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm" event={"ID":"5664696d-245a-433b-8659-c44fd4a93e26","Type":"ContainerDied","Data":"07f292ae599191afc15c7ab5036afce0fbf8873c5230e44120a44ffab7454f40"} Nov 02 20:05:47.089497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.089135 5850 scope.go:122] "RemoveContainer" containerID="5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2" Nov 02 20:05:47.090952 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.090927 5850 generic.go:356] "Generic (PLEG): container finished" podID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerID="0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f" exitCode=0 Nov 02 20:05:47.091091 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.090971 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" event={"ID":"f1fef2ff-3579-43ff-b49b-d86e0d0698e0","Type":"ContainerDied","Data":"0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f"} Nov 02 20:05:47.091091 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.091000 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" event={"ID":"f1fef2ff-3579-43ff-b49b-d86e0d0698e0","Type":"ContainerDied","Data":"f361ae1b8e6b0edf8e8988d09ebce052621cc25dab0ceb30fd064e0fe3f0f168"} Nov 02 20:05:47.091091 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.090973 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8" Nov 02 20:05:47.100086 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.100057 5850 scope.go:122] "RemoveContainer" containerID="5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2" Nov 02 20:05:47.100567 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:47.100468 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2\": container with ID starting with 5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2 not found: ID does not exist" containerID="5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2" Nov 02 20:05:47.100688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.100579 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2"} err="failed to get container status \"5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2\": rpc error: code = NotFound desc = could not find container \"5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2\": container with ID starting with 5938bac625541deef2d433db10de523bb9007eb4a1fced0c2486eaee674bd0f2 not found: ID does not exist" Nov 02 20:05:47.100688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.100604 5850 scope.go:122] "RemoveContainer" containerID="0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f" Nov 02 20:05:47.131527 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.111589 5850 scope.go:122] "RemoveContainer" containerID="0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f" Nov 02 20:05:47.131527 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:05:47.112537 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f\": container with ID starting with 0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f not found: ID does not exist" containerID="0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f" Nov 02 20:05:47.131527 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.112617 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f"} err="failed to get container status \"0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f\": rpc error: code = NotFound desc = could not find container \"0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f\": container with ID starting with 0c15d73e74e3bbfb71940a42eadc780ba0c5b7fb78a48fac189160346fef616f not found: ID does not exist" Nov 02 20:05:47.279291 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.279236 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm"] Nov 02 20:05:47.322432 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.322397 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/premium-w2-28623b-kserve-5ccbbf64c7-9gklm"] Nov 02 20:05:47.434431 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.434390 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8"] Nov 02 20:05:47.439683 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.439650 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-w2-28623b-kserve-5f7845b8d4-b7qm8"] Nov 02 20:05:47.850318 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.847647 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5664696d-245a-433b-8659-c44fd4a93e26" path="/var/lib/kubelet/pods/5664696d-245a-433b-8659-c44fd4a93e26/volumes" Nov 02 20:05:47.850318 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:47.848441 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" path="/var/lib/kubelet/pods/f1fef2ff-3579-43ff-b49b-d86e0d0698e0/volumes" Nov 02 20:05:50.733294 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:50.733245 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:51.287567 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:51.287494 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:51.447399 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:51.447365 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:55.111695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:55.111655 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:55.833220 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:55.833177 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" podUID="dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:59.019829 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.019796 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb"] Nov 02 20:05:59.020336 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.020028 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" podUID="2e11a43c-4a44-4d8f-ab4d-099257c7443a" containerName="maas-api" containerID="cri-o://7d69d7bdcd923290a4c10246501b3e772fa3d0f986448b866db50120b40feafd" gracePeriod=30 Nov 02 20:05:59.137636 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.137450 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:59.221095 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.221048 5850 generic.go:356] "Generic (PLEG): container finished" podID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerID="e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6" exitCode=0 Nov 02 20:05:59.221291 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.221092 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" event={"ID":"a632b5f5-8af8-4ec1-9d4d-7970df12d254","Type":"ContainerDied","Data":"e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6"} Nov 02 20:05:59.221938 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.221899 5850 scope.go:122] "RemoveContainer" containerID="e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6" Nov 02 20:05:59.228737 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.228548 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:05:59.228737 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.228634 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:05:59.228737 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.228673 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:05:59.239612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.238682 5850 generic.go:356] "Generic (PLEG): container finished" podID="2e11a43c-4a44-4d8f-ab4d-099257c7443a" containerID="7d69d7bdcd923290a4c10246501b3e772fa3d0f986448b866db50120b40feafd" exitCode=0 Nov 02 20:05:59.239612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.238775 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" event={"ID":"2e11a43c-4a44-4d8f-ab4d-099257c7443a","Type":"ContainerDied","Data":"7d69d7bdcd923290a4c10246501b3e772fa3d0f986448b866db50120b40feafd"} Nov 02 20:05:59.363940 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.363914 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:05:59.443416 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.443379 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e11a43c-4a44-4d8f-ab4d-099257c7443a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e11a43c-4a44-4d8f-ab4d-099257c7443a-openshift-service-ca\") pod \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " Nov 02 20:05:59.443634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.443433 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls\") pod \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " Nov 02 20:05:59.443634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.443540 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e11a43c-4a44-4d8f-ab4d-099257c7443a-kube-api-access-59l87\" (UniqueName: \"kubernetes.io/projected/2e11a43c-4a44-4d8f-ab4d-099257c7443a-kube-api-access-59l87\") pod \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " Nov 02 20:05:59.443634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.443631 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-metrics-certs\") pod \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\" (UID: \"2e11a43c-4a44-4d8f-ab4d-099257c7443a\") " Nov 02 20:05:59.444915 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.444717 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e11a43c-4a44-4d8f-ab4d-099257c7443a-openshift-service-ca" pod "2e11a43c-4a44-4d8f-ab4d-099257c7443a" (UID: "2e11a43c-4a44-4d8f-ab4d-099257c7443a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:59.449997 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.449960 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e11a43c-4a44-4d8f-ab4d-099257c7443a-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:59.465775 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.464695 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls" pod "2e11a43c-4a44-4d8f-ab4d-099257c7443a" (UID: "2e11a43c-4a44-4d8f-ab4d-099257c7443a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:59.475020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.474969 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-metrics-certs" pod "2e11a43c-4a44-4d8f-ab4d-099257c7443a" (UID: "2e11a43c-4a44-4d8f-ab4d-099257c7443a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:59.475192 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.475054 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e11a43c-4a44-4d8f-ab4d-099257c7443a-kube-api-access-59l87" pod "2e11a43c-4a44-4d8f-ab4d-099257c7443a" (UID: "2e11a43c-4a44-4d8f-ab4d-099257c7443a"). InnerVolumeSpecName "kube-api-access-59l87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:59.564712 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.564669 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:59.576437 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.576223 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59l87\" (UniqueName: \"kubernetes.io/projected/2e11a43c-4a44-4d8f-ab4d-099257c7443a-kube-api-access-59l87\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:59.576750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:05:59.576725 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e11a43c-4a44-4d8f-ab4d-099257c7443a-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:00.247366 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.247321 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" event={"ID":"a632b5f5-8af8-4ec1-9d4d-7970df12d254","Type":"ContainerStarted","Data":"fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db"} Nov 02 20:06:00.248898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.248871 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" event={"ID":"2e11a43c-4a44-4d8f-ab4d-099257c7443a","Type":"ContainerDied","Data":"90af404e348f14e5cc42f1a5a0f50c8a8cf9a0a1d76457066168dd57fc0f8548"} Nov 02 20:06:00.249035 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.248923 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb" Nov 02 20:06:00.249035 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.248926 5850 scope.go:122] "RemoveContainer" containerID="7d69d7bdcd923290a4c10246501b3e772fa3d0f986448b866db50120b40feafd" Nov 02 20:06:00.483832 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.483793 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb"] Nov 02 20:06:00.489751 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.489718 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c60a0d-78cff548c6-gmrwb"] Nov 02 20:06:00.662536 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.662469 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:06:00.662725 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.662685 5850 patch_prober.go:29] interesting pod/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:06:00.662774 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.662742 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 20:06:00.729499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.729105 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:00.996557 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:00.996446 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:01.158233 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:01.158197 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:01.662888 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:01.662852 5850 patch_prober.go:29] interesting pod/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:06:01.663313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:01.662907 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 20:06:01.732264 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:01.732217 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:01.838970 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:01.838936 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e11a43c-4a44-4d8f-ab4d-099257c7443a" path="/var/lib/kubelet/pods/2e11a43c-4a44-4d8f-ab4d-099257c7443a/volumes" Nov 02 20:06:02.662576 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:02.662543 5850 patch_prober.go:29] interesting pod/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:06:02.662763 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:02.662624 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 20:06:03.662427 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:03.662389 5850 patch_prober.go:29] interesting pod/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:06:03.662896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:03.662458 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 20:06:04.280028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.279992 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:04.662850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.662811 5850 patch_prober.go:29] interesting pod/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:06:04.663323 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.662872 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 20:06:04.733407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.733369 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:04.739594 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.739570 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 20:06:04.773830 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.773793 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-plugins-config-volume\") pod \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\" (UID: \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\") " Nov 02 20:06:04.774018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.773920 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-kube-api-access-m2td7\" (UniqueName: \"kubernetes.io/projected/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-kube-api-access-m2td7\") pod \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\" (UID: \"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02\") " Nov 02 20:06:04.774204 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.774177 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-plugins-config-volume" pod "dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" (UID: "dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:04.774392 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.774378 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:04.776300 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.776274 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-kube-api-access-m2td7" pod "dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" (UID: "dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02"). InnerVolumeSpecName "kube-api-access-m2td7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:04.875755 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:04.875705 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2td7\" (UniqueName: \"kubernetes.io/projected/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02-kube-api-access-m2td7\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:05.327547 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.322228 5850 generic.go:356] "Generic (PLEG): container finished" podID="dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" containerID="fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed" exitCode=0 Nov 02 20:06:05.327547 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.322419 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" event={"ID":"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02","Type":"ContainerDied","Data":"fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed"} Nov 02 20:06:05.327547 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.322491 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" event={"ID":"dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02","Type":"ContainerDied","Data":"ec1f26a651a96606f4c49d38133d0e4dcce824db973ceb793d08f23905e4ea3e"} Nov 02 20:06:05.327547 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.322532 5850 scope.go:122] "RemoveContainer" containerID="fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed" Nov 02 20:06:05.327547 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.322676 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg" Nov 02 20:06:05.364503 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.353083 5850 scope.go:122] "RemoveContainer" containerID="fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed" Nov 02 20:06:05.364503 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:06:05.353534 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed\": container with ID starting with fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed not found: ID does not exist" containerID="fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed" Nov 02 20:06:05.364503 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.353573 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed"} err="failed to get container status \"fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed\": rpc error: code = NotFound desc = could not find container \"fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed\": container with ID starting with fa84323c63d1b04efd392cf22f0ec10c0b6c837a5e56aaf28b24cefeb0aa86ed not found: ID does not exist" Nov 02 20:06:05.542439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.542394 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg"] Nov 02 20:06:05.593786 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.593701 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-28623b-7f45866479-5wtqg"] Nov 02 20:06:05.662970 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.662930 5850 patch_prober.go:29] interesting pod/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:06:05.663433 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.662991 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 20:06:05.836148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.836112 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" path="/var/lib/kubelet/pods/dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02/volumes" Nov 02 20:06:05.840407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.840371 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7"] Nov 02 20:06:05.840964 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.840950 5850 cpu_manager.go:414] "containerMap: removing container" podUID="dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" containerName="payload-pre-processing" Nov 02 20:06:05.841054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.840965 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" containerName="payload-pre-processing" Nov 02 20:06:05.841054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841003 5850 cpu_manager.go:414] "containerMap: removing container" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" Nov 02 20:06:05.841054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841009 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" Nov 02 20:06:05.841054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841021 5850 cpu_manager.go:414] "containerMap: removing container" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" Nov 02 20:06:05.841054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841026 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" Nov 02 20:06:05.841054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841034 5850 cpu_manager.go:414] "containerMap: removing container" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" Nov 02 20:06:05.841054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841039 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" Nov 02 20:06:05.841054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841047 5850 cpu_manager.go:414] "containerMap: removing container" podUID="2e11a43c-4a44-4d8f-ab4d-099257c7443a" containerName="maas-api" Nov 02 20:06:05.841054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841051 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e11a43c-4a44-4d8f-ab4d-099257c7443a" containerName="maas-api" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841061 5850 cpu_manager.go:414] "containerMap: removing container" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841069 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841081 5850 cpu_manager.go:414] "containerMap: removing container" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841087 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841100 5850 cpu_manager.go:414] "containerMap: removing container" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841105 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841217 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2079ca4-fcae-4d5e-8091-78fddd27fe84" containerName="main" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841229 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fef2ff-3579-43ff-b49b-d86e0d0698e0" containerName="main" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841235 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841241 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e11a43c-4a44-4d8f-ab4d-099257c7443a" containerName="maas-api" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841248 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90f1e76-4c42-4056-85f2-b3502e2e4359" containerName="main" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841253 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec5c949-3bd9-4ff6-a7be-76c6bf064239" containerName="istio-proxy" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841262 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="5664696d-245a-433b-8659-c44fd4a93e26" containerName="main" Nov 02 20:06:05.841448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.841271 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca4a5b1-35c5-4aa0-bb64-1f1f86f94c02" containerName="payload-pre-processing" Nov 02 20:06:05.845291 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.845240 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.904680 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.904642 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-13e739-openshift-default-dockercfg-k228c\"" Nov 02 20:06:05.905793 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.905762 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7"] Nov 02 20:06:05.944092 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.944054 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf89b2-daf1-44e0-a445-49d67c92861c-istiod-ca-cert\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.944719 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.944677 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-certs\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.945025 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.945004 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-data\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.945520 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.945468 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-envoy\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.945606 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.945553 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-socket\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.945606 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.945579 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-credential-socket\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.945718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.945632 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-podinfo\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.945718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.945704 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2gb\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-kube-api-access-sx2gb\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.946036 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.946015 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-token\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:05.974407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:05.974366 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:06.047385 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.047339 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-podinfo\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.047654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.047420 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2gb\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-kube-api-access-sx2gb\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.047654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.047606 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-token\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.047785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.047715 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf89b2-daf1-44e0-a445-49d67c92861c-istiod-ca-cert\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.047785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.047749 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-certs\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.047901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.047786 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-data\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.047901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.047825 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-envoy\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.047901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.047878 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-socket\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.048039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.047902 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-credential-socket\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.048237 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.048205 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-certs\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.048367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.048308 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-data\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.048419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.048383 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-credential-socket\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.048545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.048516 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-socket\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.048594 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.048549 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf89b2-daf1-44e0-a445-49d67c92861c-istiod-ca-cert\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.050113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.050094 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-podinfo\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.050186 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.050114 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-envoy\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.130508 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.127301 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2gb\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-kube-api-access-sx2gb\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.172851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.172262 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-token\") pod \"e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.183497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.176427 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:06.471895 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.471805 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:06.669423 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.669383 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:06:06.669846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.669819 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:06:06.670714 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.670691 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:06:06.725806 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.723058 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:06.904521 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.899415 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:06.971584 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.970972 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:06.971584 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.971054 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7"] Nov 02 20:06:06.971584 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.971070 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:06.971584 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:06.971114 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:07.340220 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:07.340123 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" event={"ID":"c7cf89b2-daf1-44e0-a445-49d67c92861c","Type":"ContainerStarted","Data":"8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197"} Nov 02 20:06:07.340220 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:07.340183 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" event={"ID":"c7cf89b2-daf1-44e0-a445-49d67c92861c","Type":"ContainerStarted","Data":"f3b17f7dcdce12f9109bafa6df90fbd9012c712c51f00d38c2b7d5e532321f05"} Nov 02 20:06:07.471898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:07.471863 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:07.489410 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:07.489353 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" podStartSLOduration=2.489337537 podStartE2EDuration="2.489337537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:07.486700245 +0000 UTC m=+2225.829089330" watchObservedRunningTime="2026-11-02 20:06:07.489337537 +0000 UTC m=+2225.831726617" Nov 02 20:06:07.601927 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:07.601818 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:08.478222 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:08.478190 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:08.838209 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:08.838124 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:08.996544 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:08.996507 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:09.157315 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:09.157284 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:09.391136 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:09.391100 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:09.475578 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:09.472311 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:09.481119 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:09.481083 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:06:09.542119 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:09.542083 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:09.759422 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:09.759338 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:09.919632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:09.919599 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:10.244795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:10.244766 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:10.665309 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:10.665274 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:10.875565 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:10.875519 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:11.541444 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:11.541316 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:11.968160 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:11.968125 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6"] Nov 02 20:06:11.971411 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:11.971393 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.028448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.028411 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-28fa24-openshift-default-dockercfg-t8lrb\"" Nov 02 20:06:12.033924 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.033888 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6"] Nov 02 20:06:12.069535 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.069490 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ebd778-efab-4d94-b73a-d00738bd2659-istio-podinfo\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.069713 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.069585 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-data\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.069713 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.069635 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ebd778-efab-4d94-b73a-d00738bd2659-istiod-ca-cert\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.069836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.069739 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-socket\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.069836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.069785 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-certs\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.069836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.069804 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-credential-socket\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.069982 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.069915 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzb7z\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-kube-api-access-vzb7z\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.069982 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.069945 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-istio-token\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.069982 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.069976 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-envoy\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.083935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.083903 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.193557 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ebd778-efab-4d94-b73a-d00738bd2659-istiod-ca-cert\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.193640 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-socket\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.193699 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-certs\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.193731 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-credential-socket\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.193774 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzb7z\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-kube-api-access-vzb7z\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.193811 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-istio-token\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.193869 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-envoy\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.194054 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ebd778-efab-4d94-b73a-d00738bd2659-istio-podinfo\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.194145 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-data\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.194533 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-credential-socket\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.194777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.194578 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-data\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.195525 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.195204 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ebd778-efab-4d94-b73a-d00738bd2659-istiod-ca-cert\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.195525 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.195496 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-socket\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.207089 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.207054 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-certs\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.207824 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.207699 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-envoy\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.209702 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.209651 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ebd778-efab-4d94-b73a-d00738bd2659-istio-podinfo\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.253876 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.253798 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-istio-token\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.254014 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.253954 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzb7z\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-kube-api-access-vzb7z\") pod \"e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.285984 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.285947 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:12.416287 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.416243 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:12.708084 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:06:12.708051 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ebd778_efab_4d94_b73a_d00738bd2659.slice/crio-513f528d2f140449866846b4fcb2bee75c44a0066cfd74147f234b9f518c47f6 WatchSource:0}: Error finding container 513f528d2f140449866846b4fcb2bee75c44a0066cfd74147f234b9f518c47f6: Status 404 returned error can't find the container with id 513f528d2f140449866846b4fcb2bee75c44a0066cfd74147f234b9f518c47f6 Nov 02 20:06:12.710683 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.710657 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6"] Nov 02 20:06:12.710779 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.710684 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:12.710779 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.710753 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:12.710849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.710784 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:12.779829 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:12.779794 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:13.208925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:13.208893 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:13.511507 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:13.511387 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" event={"ID":"27ebd778-efab-4d94-b73a-d00738bd2659","Type":"ContainerStarted","Data":"2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078"} Nov 02 20:06:13.511507 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:13.511446 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" event={"ID":"27ebd778-efab-4d94-b73a-d00738bd2659","Type":"ContainerStarted","Data":"513f528d2f140449866846b4fcb2bee75c44a0066cfd74147f234b9f518c47f6"} Nov 02 20:06:13.706694 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:13.706630 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" podStartSLOduration=2.70661294 podStartE2EDuration="2.70661294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:13.652833089 +0000 UTC m=+2231.995222169" watchObservedRunningTime="2026-11-02 20:06:13.70661294 +0000 UTC m=+2232.049002019" Nov 02 20:06:14.142964 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.142927 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c"] Nov 02 20:06:14.146874 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.146848 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.200832 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.200788 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-19d134-openshift-default-dockercfg-7xrgh\"" Nov 02 20:06:14.230223 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.230183 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c"] Nov 02 20:06:14.245702 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.245667 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-token\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.245885 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.245729 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-data\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.245963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.245872 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-credential-socket\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.246018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.245970 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgm2h\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-kube-api-access-rgm2h\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.246081 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.246059 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-socket\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.246149 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.246135 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-envoy\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.246270 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.246254 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-certs\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.246339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.246314 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50407317-673e-4dfa-9ee2-b70a95b97e31-istiod-ca-cert\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.246408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.246383 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-podinfo\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.306016 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.305985 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:14.309337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.309299 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:14.339499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.327264 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:14.361332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.356399 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-data\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.361332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.356518 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-credential-socket\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.361332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.356590 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgm2h\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-kube-api-access-rgm2h\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.361332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.356650 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-socket\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.361332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.356736 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-envoy\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.361332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.356832 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-certs\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.361332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.356891 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50407317-673e-4dfa-9ee2-b70a95b97e31-istiod-ca-cert\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.361332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.356920 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-podinfo\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.361332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.356962 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-token\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.371531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.370207 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50407317-673e-4dfa-9ee2-b70a95b97e31-istiod-ca-cert\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.371531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.370818 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-data\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.371531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.371396 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-certs\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.371828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.371616 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-socket\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.371889 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.371856 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-credential-socket\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.373880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.373848 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-podinfo\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.374792 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.374756 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-envoy\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.421060 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.420984 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-token\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.421884 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.421865 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgm2h\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-kube-api-access-rgm2h\") pod \"e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.462846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.462810 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:14.526603 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.526491 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:14.533303 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.532837 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:06:14.578816 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.578726 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:14.662996 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.662961 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm"] Nov 02 20:06:14.763540 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:06:14.763446 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:06:14.835192 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.835163 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:14.896013 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:06:14.895982 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50407317_673e_4dfa_9ee2_b70a95b97e31.slice/crio-a7bee769722b52d2dbab0fd16d4937fab7cd493681451e68a4b7302747fd74c5 WatchSource:0}: Error finding container a7bee769722b52d2dbab0fd16d4937fab7cd493681451e68a4b7302747fd74c5: Status 404 returned error can't find the container with id a7bee769722b52d2dbab0fd16d4937fab7cd493681451e68a4b7302747fd74c5 Nov 02 20:06:14.898312 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.898281 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c"] Nov 02 20:06:14.899403 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.899360 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:14.899465 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.899448 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:14.899550 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:14.899496 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:15.021730 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:15.021611 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:15.532550 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:15.532510 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" event={"ID":"50407317-673e-4dfa-9ee2-b70a95b97e31","Type":"ContainerStarted","Data":"401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598"} Nov 02 20:06:15.533077 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:15.532574 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" event={"ID":"50407317-673e-4dfa-9ee2-b70a95b97e31","Type":"ContainerStarted","Data":"a7bee769722b52d2dbab0fd16d4937fab7cd493681451e68a4b7302747fd74c5"} Nov 02 20:06:15.654408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:15.654345 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" podStartSLOduration=1.654325898 podStartE2EDuration="1.654325898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:15.653583729 +0000 UTC m=+2233.995972808" watchObservedRunningTime="2026-11-02 20:06:15.654325898 +0000 UTC m=+2233.996714980" Nov 02 20:06:15.696134 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:15.696094 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:15.854160 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:15.854069 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:16.405572 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:16.405538 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:16.463374 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:16.463334 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:16.469509 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:16.469465 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:16.539557 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:16.539514 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:16.540523 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:16.540503 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:06:16.565123 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:16.565088 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:16.734190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:16.734108 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:19.246919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:19.246882 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:19.417797 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:19.417763 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:19.583718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:19.583635 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:19.959579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:19.959542 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:20.140495 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:20.125319 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:21.493714 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:21.493676 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:21.738368 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:21.738328 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:21.986012 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:21.985972 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:23.433289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:23.433206 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:23.609801 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:23.609764 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:23.807661 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:23.807558 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:23.988122 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:23.988089 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:24.141559 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:24.141527 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:26.405954 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:26.405910 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:29.692877 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:29.692831 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" containerID="cri-o://41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44" gracePeriod=45 Nov 02 20:06:29.966638 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:29.966604 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 20:06:30.070552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.070464 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tmp-dir\") pod \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " Nov 02 20:06:30.070761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.070604 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tls-certs\") pod \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " Nov 02 20:06:30.070761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.070721 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-home\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-home\") pod \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " Nov 02 20:06:30.070855 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.070810 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-model-cache\") pod \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " Nov 02 20:06:30.070855 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.070817 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tmp-dir" pod "f446dc3b-8567-47c7-91b1-5291aa4fb4f0" (UID: "f446dc3b-8567-47c7-91b1-5291aa4fb4f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:06:30.070977 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.070914 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-kube-api-access-rtsmd\" (UniqueName: \"kubernetes.io/projected/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-kube-api-access-rtsmd\") pod \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " Nov 02 20:06:30.071035 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.070985 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-dshm\") pod \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\" (UID: \"f446dc3b-8567-47c7-91b1-5291aa4fb4f0\") " Nov 02 20:06:30.071094 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.071044 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-home" pod "f446dc3b-8567-47c7-91b1-5291aa4fb4f0" (UID: "f446dc3b-8567-47c7-91b1-5291aa4fb4f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:06:30.071248 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.071225 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-model-cache" pod "f446dc3b-8567-47c7-91b1-5291aa4fb4f0" (UID: "f446dc3b-8567-47c7-91b1-5291aa4fb4f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:06:30.071555 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.071532 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.071555 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.071556 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.071728 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.071570 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.073989 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.073956 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-kube-api-access-rtsmd" pod "f446dc3b-8567-47c7-91b1-5291aa4fb4f0" (UID: "f446dc3b-8567-47c7-91b1-5291aa4fb4f0"). InnerVolumeSpecName "kube-api-access-rtsmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:30.074101 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.074007 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tls-certs" pod "f446dc3b-8567-47c7-91b1-5291aa4fb4f0" (UID: "f446dc3b-8567-47c7-91b1-5291aa4fb4f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:06:30.074155 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.074126 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-dshm" pod "f446dc3b-8567-47c7-91b1-5291aa4fb4f0" (UID: "f446dc3b-8567-47c7-91b1-5291aa4fb4f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:06:30.173240 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.173195 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtsmd\" (UniqueName: \"kubernetes.io/projected/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-kube-api-access-rtsmd\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.173240 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.173242 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.173518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.173255 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f446dc3b-8567-47c7-91b1-5291aa4fb4f0-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.179940 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.179901 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:30.432763 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.432726 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:30.676627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.676588 5850 generic.go:356] "Generic (PLEG): container finished" podID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerID="41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44" exitCode=0 Nov 02 20:06:30.676811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.676637 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" Nov 02 20:06:30.676811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.676673 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" event={"ID":"f446dc3b-8567-47c7-91b1-5291aa4fb4f0","Type":"ContainerDied","Data":"41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44"} Nov 02 20:06:30.676811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.676721 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm" event={"ID":"f446dc3b-8567-47c7-91b1-5291aa4fb4f0","Type":"ContainerDied","Data":"0a8bfc0d635a8244d93a05e2e44d4b07914702acfa44d42c398a1771c5101184"} Nov 02 20:06:30.676811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.676742 5850 scope.go:122] "RemoveContainer" containerID="41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44" Nov 02 20:06:30.687824 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.687797 5850 scope.go:122] "RemoveContainer" containerID="41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44" Nov 02 20:06:30.688141 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:06:30.688117 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44\": container with ID starting with 41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44 not found: ID does not exist" containerID="41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44" Nov 02 20:06:30.688215 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.688149 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44"} err="failed to get container status \"41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44\": rpc error: code = NotFound desc = could not find container \"41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44\": container with ID starting with 41593525fe964e33c1627c11cfa1fb13c5d37a8c845ff37e695aebd64ae85d44 not found: ID does not exist" Nov 02 20:06:30.879498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.872127 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm"] Nov 02 20:06:30.918795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.918752 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/premium-w4-8795bf-kserve-c6d7c7f4c-j92bm"] Nov 02 20:06:30.927893 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.927810 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c"] Nov 02 20:06:30.929137 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.929108 5850 cpu_manager.go:414] "containerMap: removing container" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" Nov 02 20:06:30.929316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.929300 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" Nov 02 20:06:30.929682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.929653 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" containerName="main" Nov 02 20:06:30.935307 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.935242 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:30.960667 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.960624 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c"] Nov 02 20:06:30.992336 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.992289 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19d134\"" Nov 02 20:06:30.992336 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:30.992313 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19d134-dockercfg-v2xsk\"" Nov 02 20:06:31.041146 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.041104 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c\" (UID: \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:31.041413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.041391 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx24g\" (UniqueName: \"kubernetes.io/projected/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-kube-api-access-gx24g\") pod \"payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c\" (UID: \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:31.051182 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.051150 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:31.142770 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.142735 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx24g\" (UniqueName: \"kubernetes.io/projected/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-kube-api-access-gx24g\") pod \"payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c\" (UID: \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:31.142988 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.142816 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c\" (UID: \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:31.143525 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.143505 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c\" (UID: \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:31.201975 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.201899 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx24g\" (UniqueName: \"kubernetes.io/projected/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-kube-api-access-gx24g\") pod \"payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c\" (UID: \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:31.254121 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.254079 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:31.629527 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:06:31.629131 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbff140_4b0c_492c_a2ba_7163dd68b5c3.slice/crio-1612ffe15bee9ad16087d2c5a270b1a9f0c5cf4a38aaf7e0adcca81042417f46 WatchSource:0}: Error finding container 1612ffe15bee9ad16087d2c5a270b1a9f0c5cf4a38aaf7e0adcca81042417f46: Status 404 returned error can't find the container with id 1612ffe15bee9ad16087d2c5a270b1a9f0c5cf4a38aaf7e0adcca81042417f46 Nov 02 20:06:31.631981 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.631952 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c"] Nov 02 20:06:31.684550 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.684511 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" event={"ID":"9bbff140-4b0c-492c-a2ba-7163dd68b5c3","Type":"ContainerStarted","Data":"1612ffe15bee9ad16087d2c5a270b1a9f0c5cf4a38aaf7e0adcca81042417f46"} Nov 02 20:06:31.816805 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:31.816772 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f446dc3b-8567-47c7-91b1-5291aa4fb4f0" path="/var/lib/kubelet/pods/f446dc3b-8567-47c7-91b1-5291aa4fb4f0/volumes" Nov 02 20:06:32.694165 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:32.694130 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" event={"ID":"9bbff140-4b0c-492c-a2ba-7163dd68b5c3","Type":"ContainerStarted","Data":"c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9"} Nov 02 20:06:32.694638 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:32.694196 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:32.812256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:32.812204 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" podStartSLOduration=2.81219172 podStartE2EDuration="2.81219172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:32.810343348 +0000 UTC m=+2251.152732428" watchObservedRunningTime="2026-11-02 20:06:32.81219172 +0000 UTC m=+2251.154580778" Nov 02 20:06:32.896720 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:32.896687 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:33.113254 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:33.111718 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:35.453135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:35.453094 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:39.220531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:39.215191 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:39.933596 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:39.933558 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:42.534167 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:42.534125 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:42.681771 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:42.681739 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:43.705909 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:43.705879 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:06:44.326214 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:44.326075 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:44.764181 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:06:44.764137 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:06:45.594424 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:45.594337 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:45.754988 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:45.754950 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:48.739349 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:48.739317 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:48.817881 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:48.817847 5850 generic.go:356] "Generic (PLEG): container finished" podID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerID="256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317" exitCode=0 Nov 02 20:06:48.818086 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:48.817902 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" event={"ID":"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0","Type":"ContainerDied","Data":"256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317"} Nov 02 20:06:48.818393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:48.818373 5850 scope.go:122] "RemoveContainer" containerID="256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317" Nov 02 20:06:48.820221 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:48.820188 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:48.820290 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:48.820265 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:48.820327 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:48.820298 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:49.788080 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:49.788021 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:49.825451 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:49.825414 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" event={"ID":"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0","Type":"ContainerStarted","Data":"a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce"} Nov 02 20:06:49.938822 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:49.938790 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:49.945094 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:49.945070 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:06:49.945283 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:49.945257 5850 patch_prober.go:29] interesting pod/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:06:49.945346 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:49.945323 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 20:06:50.945301 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:50.945263 5850 patch_prober.go:29] interesting pod/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:06:50.945764 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:50.945323 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 20:06:51.755454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:51.755426 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:51.934343 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:51.934308 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:51.945219 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:51.945176 5850 patch_prober.go:29] interesting pod/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:06:51.945467 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:51.945252 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 20:06:52.481806 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:52.481774 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:52.945549 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:52.945499 5850 patch_prober.go:29] interesting pod/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:06:52.945946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:52.945564 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 20:06:53.238218 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:53.238127 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:53.945770 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:53.945736 5850 patch_prober.go:29] interesting pod/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:06:53.946262 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:53.945787 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 20:06:54.945981 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:54.945941 5850 patch_prober.go:29] interesting pod/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:06:54.946418 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:54.946040 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 20:06:55.950626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:55.950595 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:06:55.951100 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:55.950967 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:06:55.951688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:55.951671 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:06:56.389383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.389344 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc"] Nov 02 20:06:56.393600 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.393579 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.469368 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.469322 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba54f0b-65f3-4e15-beda-324921802c6f-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.469591 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.469377 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.469591 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.469441 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.469725 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.469660 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.469785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.469729 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.469785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.469758 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.469785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.469777 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jqt\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-kube-api-access-s9jqt\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.469949 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.469819 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.469949 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.469848 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba54f0b-65f3-4e15-beda-324921802c6f-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.481884 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.481842 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc"] Nov 02 20:06:56.571828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.571787 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba54f0b-65f3-4e15-beda-324921802c6f-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.571828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.571831 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.571856 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.571892 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.571936 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.571955 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.571980 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jqt\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-kube-api-access-s9jqt\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.572031 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.572060 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba54f0b-65f3-4e15-beda-324921802c6f-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572551 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.572265 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572621 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.572605 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572680 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.572655 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572728 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.572691 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.572981 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.572964 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba54f0b-65f3-4e15-beda-324921802c6f-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.574362 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.574334 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.574503 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.574441 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba54f0b-65f3-4e15-beda-324921802c6f-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.633036 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.633000 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.633257 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.633229 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jqt\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-kube-api-access-s9jqt\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:56.708383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:56.708297 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:57.150421 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:06:57.150378 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba54f0b_65f3_4e15_beda_324921802c6f.slice/crio-ba4730a984fe230aff2a4edaa7121f038fe442f0c12932da98520d30630cbc73 WatchSource:0}: Error finding container ba4730a984fe230aff2a4edaa7121f038fe442f0c12932da98520d30630cbc73: Status 404 returned error can't find the container with id ba4730a984fe230aff2a4edaa7121f038fe442f0c12932da98520d30630cbc73 Nov 02 20:06:57.152284 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:57.152258 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc"] Nov 02 20:06:57.152619 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:57.152586 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:57.152673 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:57.152652 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:57.152709 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:57.152682 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:06:57.652901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:57.652870 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:57.895400 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:57.895366 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:57.951231 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:57.951144 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" event={"ID":"dba54f0b-65f3-4e15-beda-324921802c6f","Type":"ContainerStarted","Data":"024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832"} Nov 02 20:06:57.951231 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:57.951230 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" event={"ID":"dba54f0b-65f3-4e15-beda-324921802c6f","Type":"ContainerStarted","Data":"ba4730a984fe230aff2a4edaa7121f038fe442f0c12932da98520d30630cbc73"} Nov 02 20:06:58.072313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:58.072253 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" podStartSLOduration=2.072238691 podStartE2EDuration="2.072238691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:58.06959849 +0000 UTC m=+2276.411987574" watchObservedRunningTime="2026-11-02 20:06:58.072238691 +0000 UTC m=+2276.414627771" Nov 02 20:06:58.727500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:58.724994 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:06:58.731647 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:58.731611 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:06:58.731925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:58.731897 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 02 20:06:59.720498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:59.710944 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:06:59.720498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:06:59.711026 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 02 20:07:00.727497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:00.726977 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:07:00.727497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:00.727047 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 02 20:07:01.454759 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:01.454722 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:01.722563 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:01.719923 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:07:01.722563 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:01.719987 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 02 20:07:02.395448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:02.395343 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:02.548027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:02.547988 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:02.726565 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:02.715976 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:07:02.726565 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:02.716042 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 02 20:07:02.934136 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:02.934107 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:03.135602 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:03.135567 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:03.304121 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:03.304095 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:03.709047 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:03.709015 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:07:03.709466 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:03.709074 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Nov 02 20:07:03.821771 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:03.821738 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:04.714305 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:04.714272 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:07:04.714778 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:04.714695 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:07:04.715429 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:04.715415 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:07:07.139316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:07.139284 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:08.705986 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:08.705954 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:08.955228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:08.955197 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:09.110729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:09.110650 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:09.595451 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:09.595410 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:09.756763 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:09.756725 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:09.923231 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:09.923198 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:12.632197 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:12.632151 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:13.722142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:13.722109 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:14.763271 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:07:14.763221 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:07:15.533858 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:15.533827 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:15.688115 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:15.688084 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:15.870898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:15.870865 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:16.234001 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:16.233909 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:16.720361 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:16.720327 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:18.401571 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:18.401537 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:18.558812 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:18.558783 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:18.710822 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:18.710715 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:21.606135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:21.606098 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:21.811241 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:21.811200 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:23.133702 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:23.133669 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:27.400222 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:27.400191 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:27.551877 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:27.551846 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:27.718449 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:27.718350 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:31.672725 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.672692 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p"] Nov 02 20:07:31.677208 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.677181 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.734333 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.734299 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p"] Nov 02 20:07:31.781925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.781890 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-13e739-serving-cert\"" Nov 02 20:07:31.824171 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.824114 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-metrics-certs\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.824419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.824286 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpd6b\" (UniqueName: \"kubernetes.io/projected/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-kube-api-access-hpd6b\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.824419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.824361 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.824590 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.824426 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.879668 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.879633 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:31.933699 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.933603 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.933891 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.933778 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-metrics-certs\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.933952 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.933926 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpd6b\" (UniqueName: \"kubernetes.io/projected/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-kube-api-access-hpd6b\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.934212 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.934004 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.939376 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.934871 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.990549 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.978143 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-metrics-certs\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:31.996528 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:31.996495 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:32.018824 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:32.018793 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpd6b\" (UniqueName: \"kubernetes.io/projected/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-kube-api-access-hpd6b\") pod \"maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:32.032059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:32.032025 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:32.180295 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:32.180262 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:32.290810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:32.290717 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:32.665551 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:07:32.665518 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfcd7fe_218c_4efd_ba81_45f03d929f4b.slice/crio-79216209370b46ff4c640573f01c62ee38b809839ea21124f9920ae9752f101f WatchSource:0}: Error finding container 79216209370b46ff4c640573f01c62ee38b809839ea21124f9920ae9752f101f: Status 404 returned error can't find the container with id 79216209370b46ff4c640573f01c62ee38b809839ea21124f9920ae9752f101f Nov 02 20:07:32.666934 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:32.666905 5850 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:07:32.668505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:32.668484 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p"] Nov 02 20:07:33.304729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:33.304693 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" event={"ID":"ddfcd7fe-218c-4efd-ba81-45f03d929f4b","Type":"ContainerStarted","Data":"79216209370b46ff4c640573f01c62ee38b809839ea21124f9920ae9752f101f"} Nov 02 20:07:34.312383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:34.312343 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" event={"ID":"ddfcd7fe-218c-4efd-ba81-45f03d929f4b","Type":"ContainerStarted","Data":"fd047667a6f3f31c51c53419f093d50ecd35d9ef76f94f7d9669b72e5736955d"} Nov 02 20:07:34.312907 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:34.312588 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:34.430284 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:34.430225 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" podStartSLOduration=2.548316326 podStartE2EDuration="3.430210421s" totalImagesPullingTime="881.894095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:34.429451842 +0000 UTC m=+2312.771840922" watchObservedRunningTime="2026-11-02 20:07:34.430210421 +0000 UTC m=+2312.772599501" Nov 02 20:07:36.433249 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:36.433215 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:40.325795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:40.325759 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:07:40.856970 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:40.856937 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:41.576959 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:41.576930 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:42.365075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:42.365034 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:43.956953 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:43.956911 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:44.123491 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:44.119203 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:44.762968 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:07:44.762924 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:07:45.796945 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:45.796898 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:50.349098 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:50.349058 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:54.841173 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:54.840980 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:57.965980 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:57.965940 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:58.144152 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:58.144118 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:58.319271 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:58.319128 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:59.481556 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:07:59.481523 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:01.234015 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:01.233961 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:01.429495 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:01.424807 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:02.594743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:02.594689 5850 patch_prober.go:29] interesting pod/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:08:02.595246 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:02.594810 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:08:04.221586 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:04.221542 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:05.672591 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:05.672510 5850 generic.go:356] "Generic (PLEG): container finished" podID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerID="401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598" exitCode=0 Nov 02 20:08:05.673634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:05.672591 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" event={"ID":"50407317-673e-4dfa-9ee2-b70a95b97e31","Type":"ContainerDied","Data":"401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598"} Nov 02 20:08:05.673634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:05.673518 5850 scope.go:122] "RemoveContainer" containerID="401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598" Nov 02 20:08:05.676076 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:05.676039 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:05.676190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:05.676122 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:05.676190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:05.676156 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:06.681857 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.681821 5850 generic.go:356] "Generic (PLEG): container finished" podID="27ebd778-efab-4d94-b73a-d00738bd2659" containerID="2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078" exitCode=0 Nov 02 20:08:06.682295 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.681920 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" event={"ID":"27ebd778-efab-4d94-b73a-d00738bd2659","Type":"ContainerDied","Data":"2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078"} Nov 02 20:08:06.682394 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.682376 5850 scope.go:122] "RemoveContainer" containerID="2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078" Nov 02 20:08:06.684340 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.684295 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:06.684464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.684400 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:06.684464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.684453 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:06.685364 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.685338 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" event={"ID":"50407317-673e-4dfa-9ee2-b70a95b97e31","Type":"ContainerStarted","Data":"fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871"} Nov 02 20:08:06.687201 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.687177 5850 generic.go:356] "Generic (PLEG): container finished" podID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerID="8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197" exitCode=0 Nov 02 20:08:06.687310 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.687207 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" event={"ID":"c7cf89b2-daf1-44e0-a445-49d67c92861c","Type":"ContainerDied","Data":"8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197"} Nov 02 20:08:06.687577 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.687562 5850 scope.go:122] "RemoveContainer" containerID="8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197" Nov 02 20:08:06.689093 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.689062 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:06.689171 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.689157 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:06.689209 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:06.689194 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:08:07.467499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:07.466947 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:08:07.467499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:07.467122 5850 patch_prober.go:29] interesting pod/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:08:07.467499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:07.467172 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 20:08:07.695963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:07.695921 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" event={"ID":"c7cf89b2-daf1-44e0-a445-49d67c92861c","Type":"ContainerStarted","Data":"1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb"} Nov 02 20:08:07.698309 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:07.698285 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" event={"ID":"27ebd778-efab-4d94-b73a-d00738bd2659","Type":"ContainerStarted","Data":"e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f"} Nov 02 20:08:08.286337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:08.286286 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:08:08.286337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:08.286341 5850 patch_prober.go:29] interesting pod/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:08:08.286663 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:08.286386 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 20:08:08.466498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:08.465899 5850 patch_prober.go:29] interesting pod/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:08:08.466498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:08.465964 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 20:08:08.483501 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:08.480295 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:08:08.484656 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:08.484620 5850 patch_prober.go:29] interesting pod/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:08:08.484786 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:08.484691 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 20:08:08.654498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:08.653000 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:09.287222 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:09.287180 5850 patch_prober.go:29] interesting pod/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:08:09.287774 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:09.287250 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 20:08:09.467141 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:09.467038 5850 patch_prober.go:29] interesting pod/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:08:09.467447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:09.467399 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 20:08:09.474650 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:09.474604 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:08:09.475334 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:09.475307 5850 patch_prober.go:29] interesting pod/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:08:09.475553 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:09.475514 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 20:08:10.287655 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:10.287606 5850 patch_prober.go:29] interesting pod/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:08:10.288075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:10.287668 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 20:08:10.467497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:10.466921 5850 patch_prober.go:29] interesting pod/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:08:10.467497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:10.466990 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 20:08:10.482534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:10.479880 5850 patch_prober.go:29] interesting pod/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:08:10.482534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:10.479955 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 20:08:11.286277 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:11.286236 5850 patch_prober.go:29] interesting pod/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:08:11.286550 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:11.286289 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 20:08:11.475499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:11.474963 5850 patch_prober.go:29] interesting pod/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:08:11.475499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:11.475034 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 20:08:11.478419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:11.478385 5850 patch_prober.go:29] interesting pod/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:08:11.478926 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:11.478894 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 20:08:12.287340 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:12.287297 5850 patch_prober.go:29] interesting pod/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:08:12.287560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:12.287358 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 20:08:12.468170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:12.468140 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:08:12.468646 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:12.468607 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:08:12.469090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:12.469068 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:08:12.472383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:12.472361 5850 patch_prober.go:29] interesting pod/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:08:12.472461 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:12.472426 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 20:08:13.049496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:13.043355 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:13.353915 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:13.353771 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:08:13.354574 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:13.354545 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:08:13.356790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:13.356765 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:08:13.476666 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:13.476630 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:08:13.477714 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:13.477690 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:08:14.762490 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:08:14.762440 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:08:16.550391 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:16.550356 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:17.464328 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:17.464283 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:21.808307 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:21.808276 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:26.238880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:26.238840 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:30.634800 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:30.634760 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:33.463969 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:33.463929 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:33.621408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:33.621347 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:34.354517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.354430 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4"] Nov 02 20:08:34.354761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.354733 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" podUID="309cd60f-3fb8-47fa-a67a-3248d6849fed" containerName="maas-api" containerID="cri-o://c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7" gracePeriod=30 Nov 02 20:08:34.449740 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.449554 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:34.879028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.877364 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 20:08:34.953100 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.953001 5850 generic.go:356] "Generic (PLEG): container finished" podID="309cd60f-3fb8-47fa-a67a-3248d6849fed" containerID="c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7" exitCode=0 Nov 02 20:08:34.953100 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.953061 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" Nov 02 20:08:34.953309 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.953088 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" event={"ID":"309cd60f-3fb8-47fa-a67a-3248d6849fed","Type":"ContainerDied","Data":"c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7"} Nov 02 20:08:34.953309 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.953158 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4" event={"ID":"309cd60f-3fb8-47fa-a67a-3248d6849fed","Type":"ContainerDied","Data":"153a52a8f42412c33b1eaecb7fdd2260fc5821d145486fc18b07d7532e40b16c"} Nov 02 20:08:34.953309 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.953191 5850 scope.go:122] "RemoveContainer" containerID="c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7" Nov 02 20:08:34.963455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.963432 5850 scope.go:122] "RemoveContainer" containerID="c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7" Nov 02 20:08:34.963825 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:08:34.963796 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7\": container with ID starting with c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7 not found: ID does not exist" containerID="c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7" Nov 02 20:08:34.963917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.963836 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7"} err="failed to get container status \"c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7\": rpc error: code = NotFound desc = could not find container \"c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7\": container with ID starting with c81be8665f36f0e2c6821d0d3e70d1edd76def643bbe0eb29caba702384e04c7 not found: ID does not exist" Nov 02 20:08:34.969407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.969379 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls\") pod \"309cd60f-3fb8-47fa-a67a-3248d6849fed\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " Nov 02 20:08:34.969692 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.969672 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-metrics-certs\") pod \"309cd60f-3fb8-47fa-a67a-3248d6849fed\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " Nov 02 20:08:34.969837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.969803 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309cd60f-3fb8-47fa-a67a-3248d6849fed-kube-api-access-6h577\" (UniqueName: \"kubernetes.io/projected/309cd60f-3fb8-47fa-a67a-3248d6849fed-kube-api-access-6h577\") pod \"309cd60f-3fb8-47fa-a67a-3248d6849fed\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " Nov 02 20:08:34.969907 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.969883 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309cd60f-3fb8-47fa-a67a-3248d6849fed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd60f-3fb8-47fa-a67a-3248d6849fed-openshift-service-ca\") pod \"309cd60f-3fb8-47fa-a67a-3248d6849fed\" (UID: \"309cd60f-3fb8-47fa-a67a-3248d6849fed\") " Nov 02 20:08:34.970277 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.970250 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309cd60f-3fb8-47fa-a67a-3248d6849fed-openshift-service-ca" pod "309cd60f-3fb8-47fa-a67a-3248d6849fed" (UID: "309cd60f-3fb8-47fa-a67a-3248d6849fed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:34.970555 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.970535 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309cd60f-3fb8-47fa-a67a-3248d6849fed-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:34.971729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.971707 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls" pod "309cd60f-3fb8-47fa-a67a-3248d6849fed" (UID: "309cd60f-3fb8-47fa-a67a-3248d6849fed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:08:34.971818 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.971772 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-metrics-certs" pod "309cd60f-3fb8-47fa-a67a-3248d6849fed" (UID: "309cd60f-3fb8-47fa-a67a-3248d6849fed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:08:34.971878 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:34.971865 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309cd60f-3fb8-47fa-a67a-3248d6849fed-kube-api-access-6h577" pod "309cd60f-3fb8-47fa-a67a-3248d6849fed" (UID: "309cd60f-3fb8-47fa-a67a-3248d6849fed"). InnerVolumeSpecName "kube-api-access-6h577". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:35.072151 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:35.072109 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:35.072151 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:35.072148 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h577\" (UniqueName: \"kubernetes.io/projected/309cd60f-3fb8-47fa-a67a-3248d6849fed-kube-api-access-6h577\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:35.072151 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:35.072160 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309cd60f-3fb8-47fa-a67a-3248d6849fed-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:35.507466 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:35.507426 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4"] Nov 02 20:08:35.558753 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:35.558716 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8795bf-6f8f86b666-5plf4"] Nov 02 20:08:35.817523 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:35.816730 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309cd60f-3fb8-47fa-a67a-3248d6849fed" path="/var/lib/kubelet/pods/309cd60f-3fb8-47fa-a67a-3248d6849fed/volumes" Nov 02 20:08:35.863704 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:35.863625 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:36.231284 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:36.231239 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:36.402258 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:36.402216 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:36.817268 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:36.817239 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:39.258753 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:39.258715 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:40.875302 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:40.875265 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:42.759665 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:42.759630 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:42.908826 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:42.908788 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:43.078550 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:43.078446 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc"] Nov 02 20:08:43.078743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:43.078688 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" containerID="cri-o://024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832" gracePeriod=30 Nov 02 20:08:43.081537 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:43.081512 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr"] Nov 02 20:08:43.081773 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:43.081743 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" containerID="cri-o://ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954" gracePeriod=30 Nov 02 20:08:43.097380 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:43.097344 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:43.383738 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:43.383699 5850 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:08:43.383919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:43.383756 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:08:44.100500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:44.100460 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:44.807426 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:08:44.807381 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:08:45.337110 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:45.337077 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:46.434436 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:46.434401 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:48.411116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.411090 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:08:48.431676 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.431643 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 20:08:48.517034 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517001 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-certs\") pod \"dba54f0b-65f3-4e15-beda-324921802c6f\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " Nov 02 20:08:48.517238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517047 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-socket\") pod \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " Nov 02 20:08:48.517238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517068 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-kube-api-access-r2vqb\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-kube-api-access-r2vqb\") pod \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " Nov 02 20:08:48.517238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517099 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-data\") pod \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " Nov 02 20:08:48.517416 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517291 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-podinfo\") pod \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " Nov 02 20:08:48.517416 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517351 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-socket" pod "d43bb82d-4270-4943-ad7f-c16cb35f86fc" (UID: "d43bb82d-4270-4943-ad7f-c16cb35f86fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.517416 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517360 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-certs" pod "dba54f0b-65f3-4e15-beda-324921802c6f" (UID: "dba54f0b-65f3-4e15-beda-324921802c6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.517579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517366 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-credential-socket\") pod \"dba54f0b-65f3-4e15-beda-324921802c6f\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " Nov 02 20:08:48.517579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517457 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-data" pod "d43bb82d-4270-4943-ad7f-c16cb35f86fc" (UID: "d43bb82d-4270-4943-ad7f-c16cb35f86fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.517684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517593 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-envoy\") pod \"dba54f0b-65f3-4e15-beda-324921802c6f\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " Nov 02 20:08:48.517684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517611 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-credential-socket" pod "dba54f0b-65f3-4e15-beda-324921802c6f" (UID: "dba54f0b-65f3-4e15-beda-324921802c6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.517684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517673 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-data\") pod \"dba54f0b-65f3-4e15-beda-324921802c6f\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " Nov 02 20:08:48.517849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517707 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-kube-api-access-s9jqt\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-kube-api-access-s9jqt\") pod \"dba54f0b-65f3-4e15-beda-324921802c6f\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " Nov 02 20:08:48.517849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517780 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba54f0b-65f3-4e15-beda-324921802c6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba54f0b-65f3-4e15-beda-324921802c6f-istiod-ca-cert\") pod \"dba54f0b-65f3-4e15-beda-324921802c6f\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " Nov 02 20:08:48.517849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517823 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dba54f0b-65f3-4e15-beda-324921802c6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba54f0b-65f3-4e15-beda-324921802c6f-istio-podinfo\") pod \"dba54f0b-65f3-4e15-beda-324921802c6f\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " Nov 02 20:08:48.518027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517873 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-token\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-token\") pod \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " Nov 02 20:08:48.518027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517898 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-data" pod "dba54f0b-65f3-4e15-beda-324921802c6f" (UID: "dba54f0b-65f3-4e15-beda-324921802c6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.518027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517922 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-istio-token\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-istio-token\") pod \"dba54f0b-65f3-4e15-beda-324921802c6f\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " Nov 02 20:08:48.518027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.517985 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-credential-socket\") pod \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " Nov 02 20:08:48.518238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.518028 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istiod-ca-cert\") pod \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " Nov 02 20:08:48.518238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.518055 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-envoy\") pod \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " Nov 02 20:08:48.518238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.518088 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-socket\") pod \"dba54f0b-65f3-4e15-beda-324921802c6f\" (UID: \"dba54f0b-65f3-4e15-beda-324921802c6f\") " Nov 02 20:08:48.518238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.518139 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-certs\") pod \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\" (UID: \"d43bb82d-4270-4943-ad7f-c16cb35f86fc\") " Nov 02 20:08:48.518469 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.518374 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba54f0b-65f3-4e15-beda-324921802c6f-istiod-ca-cert" pod "dba54f0b-65f3-4e15-beda-324921802c6f" (UID: "dba54f0b-65f3-4e15-beda-324921802c6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:48.518802 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.518775 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istiod-ca-cert" pod "d43bb82d-4270-4943-ad7f-c16cb35f86fc" (UID: "d43bb82d-4270-4943-ad7f-c16cb35f86fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:48.518966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.518947 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.519038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.518972 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba54f0b-65f3-4e15-beda-324921802c6f-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.519038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.518989 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.519038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.519005 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.519038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.519019 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.519038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.519034 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.519321 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.519049 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.519669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.519638 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-socket" pod "dba54f0b-65f3-4e15-beda-324921802c6f" (UID: "dba54f0b-65f3-4e15-beda-324921802c6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.519785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.519679 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-certs" pod "d43bb82d-4270-4943-ad7f-c16cb35f86fc" (UID: "d43bb82d-4270-4943-ad7f-c16cb35f86fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.519785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.519734 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-credential-socket" pod "d43bb82d-4270-4943-ad7f-c16cb35f86fc" (UID: "d43bb82d-4270-4943-ad7f-c16cb35f86fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.520062 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.520017 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-envoy" pod "dba54f0b-65f3-4e15-beda-324921802c6f" (UID: "dba54f0b-65f3-4e15-beda-324921802c6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.520149 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.520082 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-podinfo" pod "d43bb82d-4270-4943-ad7f-c16cb35f86fc" (UID: "d43bb82d-4270-4943-ad7f-c16cb35f86fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:08:48.520209 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.520141 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-kube-api-access-r2vqb" pod "d43bb82d-4270-4943-ad7f-c16cb35f86fc" (UID: "d43bb82d-4270-4943-ad7f-c16cb35f86fc"). InnerVolumeSpecName "kube-api-access-r2vqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:48.520280 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.520206 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-kube-api-access-s9jqt" pod "dba54f0b-65f3-4e15-beda-324921802c6f" (UID: "dba54f0b-65f3-4e15-beda-324921802c6f"). InnerVolumeSpecName "kube-api-access-s9jqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:48.520632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.520615 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dba54f0b-65f3-4e15-beda-324921802c6f-istio-podinfo" pod "dba54f0b-65f3-4e15-beda-324921802c6f" (UID: "dba54f0b-65f3-4e15-beda-324921802c6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:08:48.521122 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.521104 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-envoy" pod "d43bb82d-4270-4943-ad7f-c16cb35f86fc" (UID: "d43bb82d-4270-4943-ad7f-c16cb35f86fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.521844 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.521826 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-token" pod "d43bb82d-4270-4943-ad7f-c16cb35f86fc" (UID: "d43bb82d-4270-4943-ad7f-c16cb35f86fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:48.521844 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.521835 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-istio-token" pod "dba54f0b-65f3-4e15-beda-324921802c6f" (UID: "dba54f0b-65f3-4e15-beda-324921802c6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:48.620224 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620185 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620224 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620222 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jqt\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-kube-api-access-s9jqt\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620237 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba54f0b-65f3-4e15-beda-324921802c6f-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620251 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620267 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba54f0b-65f3-4e15-beda-324921802c6f-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620280 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620294 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620308 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba54f0b-65f3-4e15-beda-324921802c6f-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620322 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43bb82d-4270-4943-ad7f-c16cb35f86fc-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620336 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2vqb\" (UniqueName: \"kubernetes.io/projected/d43bb82d-4270-4943-ad7f-c16cb35f86fc-kube-api-access-r2vqb\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.620517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:48.620350 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43bb82d-4270-4943-ad7f-c16cb35f86fc-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:49.053518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.051567 5850 generic.go:356] "Generic (PLEG): container finished" podID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerID="ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954" exitCode=0 Nov 02 20:08:49.053518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.051649 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" event={"ID":"d43bb82d-4270-4943-ad7f-c16cb35f86fc","Type":"ContainerDied","Data":"ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954"} Nov 02 20:08:49.053518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.051714 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" event={"ID":"d43bb82d-4270-4943-ad7f-c16cb35f86fc","Type":"ContainerDied","Data":"48d34ce1c4f975e22f6e86658c9b4b6ad2a75acfdf5f953890dc526a74b0ac2c"} Nov 02 20:08:49.053518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.051743 5850 scope.go:122] "RemoveContainer" containerID="ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954" Nov 02 20:08:49.053518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.051865 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr" Nov 02 20:08:49.054370 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.054347 5850 generic.go:356] "Generic (PLEG): container finished" podID="dba54f0b-65f3-4e15-beda-324921802c6f" containerID="024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832" exitCode=0 Nov 02 20:08:49.054452 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.054437 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" event={"ID":"dba54f0b-65f3-4e15-beda-324921802c6f","Type":"ContainerDied","Data":"024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832"} Nov 02 20:08:49.054543 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.054526 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" event={"ID":"dba54f0b-65f3-4e15-beda-324921802c6f","Type":"ContainerDied","Data":"ba4730a984fe230aff2a4edaa7121f038fe442f0c12932da98520d30630cbc73"} Nov 02 20:08:49.054623 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.054609 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc" Nov 02 20:08:49.076279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.076247 5850 scope.go:122] "RemoveContainer" containerID="ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954" Nov 02 20:08:49.076645 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:08:49.076621 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954\": container with ID starting with ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954 not found: ID does not exist" containerID="ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954" Nov 02 20:08:49.076742 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.076654 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954"} err="failed to get container status \"ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954\": rpc error: code = NotFound desc = could not find container \"ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954\": container with ID starting with ff15ca4e5407f628d0f111fe01f6d5978e47d5eadddca2e4c17767b67ab73954 not found: ID does not exist" Nov 02 20:08:49.076742 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.076675 5850 scope.go:122] "RemoveContainer" containerID="024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832" Nov 02 20:08:49.088154 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.088125 5850 scope.go:122] "RemoveContainer" containerID="024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832" Nov 02 20:08:49.088569 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:08:49.088543 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832\": container with ID starting with 024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832 not found: ID does not exist" containerID="024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832" Nov 02 20:08:49.088709 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.088574 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832"} err="failed to get container status \"024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832\": rpc error: code = NotFound desc = could not find container \"024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832\": container with ID starting with 024aa1975c217267ea3bb0810eb6335eb92ec321deea47b6e3a9d3b57045e832 not found: ID does not exist" Nov 02 20:08:49.226872 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.226839 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr"] Nov 02 20:08:49.229601 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.229570 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-7kwwr"] Nov 02 20:08:49.402167 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.402092 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc"] Nov 02 20:08:49.415498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.410930 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-khtwc"] Nov 02 20:08:49.815701 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.815627 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" path="/var/lib/kubelet/pods/d43bb82d-4270-4943-ad7f-c16cb35f86fc/volumes" Nov 02 20:08:49.816163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:49.816150 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" path="/var/lib/kubelet/pods/dba54f0b-65f3-4e15-beda-324921802c6f/volumes" Nov 02 20:08:50.132262 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:50.132225 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:53.468401 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:53.468364 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:55.006332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:55.006293 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:57.083280 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:57.083241 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp"] Nov 02 20:08:57.083793 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:57.083497 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" containerID="cri-o://2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70" gracePeriod=30 Nov 02 20:08:59.355391 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:59.355350 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:59.562180 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:59.562141 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:08:59.562351 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:08:59.562209 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:09:02.344250 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.344220 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 20:09:02.350968 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.350943 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-certs\") pod \"58ba8694-51fd-4387-85b2-808b7ebbdc93\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " Nov 02 20:09:02.351067 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.350994 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-credential-socket\") pod \"58ba8694-51fd-4387-85b2-808b7ebbdc93\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " Nov 02 20:09:02.351067 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351020 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-socket\") pod \"58ba8694-51fd-4387-85b2-808b7ebbdc93\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " Nov 02 20:09:02.351147 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351075 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-data\") pod \"58ba8694-51fd-4387-85b2-808b7ebbdc93\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " Nov 02 20:09:02.351147 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351100 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-podinfo\") pod \"58ba8694-51fd-4387-85b2-808b7ebbdc93\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " Nov 02 20:09:02.351147 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351117 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ba8694-51fd-4387-85b2-808b7ebbdc93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba8694-51fd-4387-85b2-808b7ebbdc93-istiod-ca-cert\") pod \"58ba8694-51fd-4387-85b2-808b7ebbdc93\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " Nov 02 20:09:02.351307 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351177 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-token\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-token\") pod \"58ba8694-51fd-4387-85b2-808b7ebbdc93\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " Nov 02 20:09:02.351307 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351214 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-kube-api-access-4czrh\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-kube-api-access-4czrh\") pod \"58ba8694-51fd-4387-85b2-808b7ebbdc93\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " Nov 02 20:09:02.351307 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351221 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-certs" pod "58ba8694-51fd-4387-85b2-808b7ebbdc93" (UID: "58ba8694-51fd-4387-85b2-808b7ebbdc93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:02.351307 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351284 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-envoy\") pod \"58ba8694-51fd-4387-85b2-808b7ebbdc93\" (UID: \"58ba8694-51fd-4387-85b2-808b7ebbdc93\") " Nov 02 20:09:02.351548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351331 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-socket" pod "58ba8694-51fd-4387-85b2-808b7ebbdc93" (UID: "58ba8694-51fd-4387-85b2-808b7ebbdc93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:02.351548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351349 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-credential-socket" pod "58ba8694-51fd-4387-85b2-808b7ebbdc93" (UID: "58ba8694-51fd-4387-85b2-808b7ebbdc93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:02.351548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351382 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-data" pod "58ba8694-51fd-4387-85b2-808b7ebbdc93" (UID: "58ba8694-51fd-4387-85b2-808b7ebbdc93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:02.351726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351579 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ba8694-51fd-4387-85b2-808b7ebbdc93-istiod-ca-cert" pod "58ba8694-51fd-4387-85b2-808b7ebbdc93" (UID: "58ba8694-51fd-4387-85b2-808b7ebbdc93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:02.351987 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351961 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:02.352097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.351989 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:02.352097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.352003 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:02.352097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.352017 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:02.352097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.352029 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba8694-51fd-4387-85b2-808b7ebbdc93-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:02.353468 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.353434 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-podinfo" pod "58ba8694-51fd-4387-85b2-808b7ebbdc93" (UID: "58ba8694-51fd-4387-85b2-808b7ebbdc93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:09:02.353468 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.353436 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-token" pod "58ba8694-51fd-4387-85b2-808b7ebbdc93" (UID: "58ba8694-51fd-4387-85b2-808b7ebbdc93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:02.353826 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.353804 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-envoy" pod "58ba8694-51fd-4387-85b2-808b7ebbdc93" (UID: "58ba8694-51fd-4387-85b2-808b7ebbdc93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:02.353900 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.353877 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-kube-api-access-4czrh" pod "58ba8694-51fd-4387-85b2-808b7ebbdc93" (UID: "58ba8694-51fd-4387-85b2-808b7ebbdc93"). InnerVolumeSpecName "kube-api-access-4czrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:02.453347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.453317 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:02.453347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.453346 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:02.453347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.453359 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czrh\" (UniqueName: \"kubernetes.io/projected/58ba8694-51fd-4387-85b2-808b7ebbdc93-kube-api-access-4czrh\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:02.453646 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:02.453371 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba8694-51fd-4387-85b2-808b7ebbdc93-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:03.151103 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.151063 5850 generic.go:356] "Generic (PLEG): container finished" podID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerID="2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70" exitCode=0 Nov 02 20:09:03.151266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.151136 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" Nov 02 20:09:03.151266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.151146 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" event={"ID":"58ba8694-51fd-4387-85b2-808b7ebbdc93","Type":"ContainerDied","Data":"2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70"} Nov 02 20:09:03.151266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.151205 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp" event={"ID":"58ba8694-51fd-4387-85b2-808b7ebbdc93","Type":"ContainerDied","Data":"bf96d543a9a2e06429128a5504e71e0e1b6250997b806e0144885677d0693955"} Nov 02 20:09:03.151266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.151227 5850 scope.go:122] "RemoveContainer" containerID="2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70" Nov 02 20:09:03.163324 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.163301 5850 scope.go:122] "RemoveContainer" containerID="2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70" Nov 02 20:09:03.163683 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:09:03.163661 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70\": container with ID starting with 2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70 not found: ID does not exist" containerID="2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70" Nov 02 20:09:03.163740 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.163694 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70"} err="failed to get container status \"2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70\": rpc error: code = NotFound desc = could not find container \"2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70\": container with ID starting with 2fad392f2d23d056f25b70291189ee0c4bc9acf0dce4c2286f173ca354773b70 not found: ID does not exist" Nov 02 20:09:03.375841 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.375803 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp"] Nov 02 20:09:03.427221 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.427133 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-lv6rp"] Nov 02 20:09:03.712876 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.712795 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:03.816375 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:03.816343 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" path="/var/lib/kubelet/pods/58ba8694-51fd-4387-85b2-808b7ebbdc93/volumes" Nov 02 20:09:08.127668 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:08.127636 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:12.331748 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:12.331710 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27"] Nov 02 20:09:12.332553 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:12.332520 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" containerID="cri-o://50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063" gracePeriod=30 Nov 02 20:09:12.422179 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:12.422128 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:12.593424 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:12.593325 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:12.749699 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:12.749662 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:14.762728 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:09:14.762692 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:09:16.927525 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:16.927454 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:17.082953 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.082918 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:17.365386 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.365296 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:17.639991 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.639964 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 20:09:17.760740 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.760704 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-kube-api-access-lrl2j\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-kube-api-access-lrl2j\") pod \"83b910d3-426d-4bff-8baf-7db10fb0937d\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " Nov 02 20:09:17.760920 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.760758 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-credential-socket\") pod \"83b910d3-426d-4bff-8baf-7db10fb0937d\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " Nov 02 20:09:17.760920 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.760784 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-envoy\") pod \"83b910d3-426d-4bff-8baf-7db10fb0937d\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " Nov 02 20:09:17.760920 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.760866 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-socket\") pod \"83b910d3-426d-4bff-8baf-7db10fb0937d\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " Nov 02 20:09:17.761095 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.760934 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-podinfo\") pod \"83b910d3-426d-4bff-8baf-7db10fb0937d\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " Nov 02 20:09:17.761095 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.760983 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-token\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-token\") pod \"83b910d3-426d-4bff-8baf-7db10fb0937d\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " Nov 02 20:09:17.761095 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.761041 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-certs\") pod \"83b910d3-426d-4bff-8baf-7db10fb0937d\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " Nov 02 20:09:17.761259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.761102 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-credential-socket" pod "83b910d3-426d-4bff-8baf-7db10fb0937d" (UID: "83b910d3-426d-4bff-8baf-7db10fb0937d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.761259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.761196 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-socket" pod "83b910d3-426d-4bff-8baf-7db10fb0937d" (UID: "83b910d3-426d-4bff-8baf-7db10fb0937d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.761259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.761217 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b910d3-426d-4bff-8baf-7db10fb0937d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b910d3-426d-4bff-8baf-7db10fb0937d-istiod-ca-cert\") pod \"83b910d3-426d-4bff-8baf-7db10fb0937d\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " Nov 02 20:09:17.761400 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.761270 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-data\") pod \"83b910d3-426d-4bff-8baf-7db10fb0937d\" (UID: \"83b910d3-426d-4bff-8baf-7db10fb0937d\") " Nov 02 20:09:17.761465 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.761444 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-certs" pod "83b910d3-426d-4bff-8baf-7db10fb0937d" (UID: "83b910d3-426d-4bff-8baf-7db10fb0937d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.761544 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.761497 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-data" pod "83b910d3-426d-4bff-8baf-7db10fb0937d" (UID: "83b910d3-426d-4bff-8baf-7db10fb0937d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.761544 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.761522 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b910d3-426d-4bff-8baf-7db10fb0937d-istiod-ca-cert" pod "83b910d3-426d-4bff-8baf-7db10fb0937d" (UID: "83b910d3-426d-4bff-8baf-7db10fb0937d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:17.762075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.762054 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b910d3-426d-4bff-8baf-7db10fb0937d-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.762075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.762073 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.762266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.762083 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.762266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.762092 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.762266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.762101 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.763314 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.763289 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-token" pod "83b910d3-426d-4bff-8baf-7db10fb0937d" (UID: "83b910d3-426d-4bff-8baf-7db10fb0937d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:17.763409 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.763335 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-envoy" pod "83b910d3-426d-4bff-8baf-7db10fb0937d" (UID: "83b910d3-426d-4bff-8baf-7db10fb0937d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.763459 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.763442 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-podinfo" pod "83b910d3-426d-4bff-8baf-7db10fb0937d" (UID: "83b910d3-426d-4bff-8baf-7db10fb0937d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:09:17.763534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.763490 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-kube-api-access-lrl2j" pod "83b910d3-426d-4bff-8baf-7db10fb0937d" (UID: "83b910d3-426d-4bff-8baf-7db10fb0937d"). InnerVolumeSpecName "kube-api-access-lrl2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:17.865661 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.865616 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrl2j\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-kube-api-access-lrl2j\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.865661 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.865664 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.865919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.865690 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.865919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:17.865704 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b910d3-426d-4bff-8baf-7db10fb0937d-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:18.254149 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:18.254109 5850 generic.go:356] "Generic (PLEG): container finished" podID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerID="50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063" exitCode=0 Nov 02 20:09:18.254685 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:18.254173 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" Nov 02 20:09:18.254685 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:18.254196 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" event={"ID":"83b910d3-426d-4bff-8baf-7db10fb0937d","Type":"ContainerDied","Data":"50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063"} Nov 02 20:09:18.254685 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:18.254299 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27" event={"ID":"83b910d3-426d-4bff-8baf-7db10fb0937d","Type":"ContainerDied","Data":"7b873e4d2a7f36176f22c822c09d5c61e5ea84232f500f0b33e17c1819f7352b"} Nov 02 20:09:18.254685 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:18.254339 5850 scope.go:122] "RemoveContainer" containerID="50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063" Nov 02 20:09:18.264944 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:18.264924 5850 scope.go:122] "RemoveContainer" containerID="50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063" Nov 02 20:09:18.265282 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:09:18.265257 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063\": container with ID starting with 50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063 not found: ID does not exist" containerID="50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063" Nov 02 20:09:18.265380 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:18.265291 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063"} err="failed to get container status \"50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063\": rpc error: code = NotFound desc = could not find container \"50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063\": container with ID starting with 50eee0616153a6883f359224743389dd77db8000c3f10d7fe16fcd604eb1b063 not found: ID does not exist" Nov 02 20:09:18.423794 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:18.423754 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27"] Nov 02 20:09:18.475156 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:18.475104 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nsv27"] Nov 02 20:09:19.816050 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:19.816018 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" path="/var/lib/kubelet/pods/83b910d3-426d-4bff-8baf-7db10fb0937d/volumes" Nov 02 20:09:21.352753 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:21.352720 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:25.837420 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:25.837384 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:30.180126 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:30.180092 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:33.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:33.479757 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:34.521922 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:34.521859 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:38.922942 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:38.922898 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:39.283971 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.283882 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr"] Nov 02 20:09:39.284575 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284556 5850 cpu_manager.go:414] "containerMap: removing container" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" Nov 02 20:09:39.284627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284576 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" Nov 02 20:09:39.284627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284587 5850 cpu_manager.go:414] "containerMap: removing container" podUID="309cd60f-3fb8-47fa-a67a-3248d6849fed" containerName="maas-api" Nov 02 20:09:39.284627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284593 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309cd60f-3fb8-47fa-a67a-3248d6849fed" containerName="maas-api" Nov 02 20:09:39.284627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284601 5850 cpu_manager.go:414] "containerMap: removing container" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" Nov 02 20:09:39.284627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284606 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" Nov 02 20:09:39.284787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284642 5850 cpu_manager.go:414] "containerMap: removing container" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" Nov 02 20:09:39.284787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284648 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" Nov 02 20:09:39.284787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284667 5850 cpu_manager.go:414] "containerMap: removing container" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" Nov 02 20:09:39.284787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284671 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" Nov 02 20:09:39.284787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284752 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ba8694-51fd-4387-85b2-808b7ebbdc93" containerName="istio-proxy" Nov 02 20:09:39.284787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284768 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b910d3-426d-4bff-8baf-7db10fb0937d" containerName="istio-proxy" Nov 02 20:09:39.284787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284776 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba54f0b-65f3-4e15-beda-324921802c6f" containerName="istio-proxy" Nov 02 20:09:39.284787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284786 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43bb82d-4270-4943-ad7f-c16cb35f86fc" containerName="istio-proxy" Nov 02 20:09:39.285026 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.284797 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="309cd60f-3fb8-47fa-a67a-3248d6849fed" containerName="maas-api" Nov 02 20:09:39.287664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.287644 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.345916 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.345829 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr"] Nov 02 20:09:39.378171 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.378117 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-home\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.378384 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.378225 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-model-cache\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.378384 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.378274 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpkgc\" (UniqueName: \"kubernetes.io/projected/da593eb1-73c6-417d-ba23-e979e0866e2d-kube-api-access-rpkgc\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.378574 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.378550 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-dshm\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.378650 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.378633 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-tmp-dir\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.378687 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.378679 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da593eb1-73c6-417d-ba23-e979e0866e2d-tls-certs\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.393316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.393268 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7025fa\"/\"openshift-service-ca.crt\"" Nov 02 20:09:39.393316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.393296 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7025fa\"/\"default-dockercfg-5kknb\"" Nov 02 20:09:39.393682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.393324 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7025fa\"/\"kube-root-ca.crt\"" Nov 02 20:09:39.393799 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.393698 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7025fa\"/\"test-model-7025fa-kserve-self-signed-certs\"" Nov 02 20:09:39.393799 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.393734 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:39.479921 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.479884 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-model-cache\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.480143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.479932 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpkgc\" (UniqueName: \"kubernetes.io/projected/da593eb1-73c6-417d-ba23-e979e0866e2d-kube-api-access-rpkgc\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.480143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.480054 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-dshm\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.480143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.480096 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-tmp-dir\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.480143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.480130 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da593eb1-73c6-417d-ba23-e979e0866e2d-tls-certs\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.480391 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.480189 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-home\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.480391 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.480373 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-model-cache\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.480528 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.480505 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-tmp-dir\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.480583 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.480564 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-home\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.482344 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.482327 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-dshm\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.482651 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.482635 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da593eb1-73c6-417d-ba23-e979e0866e2d-tls-certs\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.541290 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.541210 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpkgc\" (UniqueName: \"kubernetes.io/projected/da593eb1-73c6-417d-ba23-e979e0866e2d-kube-api-access-rpkgc\") pod \"test-model-7025fa-kserve-d6f649cc-bccrr\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.599565 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.599522 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:39.973766 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:09:39.973734 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda593eb1_73c6_417d_ba23_e979e0866e2d.slice/crio-ca11fa66d7238bcfc469c55c4df22c9585e170e434ce82a525ebefdce2612516 WatchSource:0}: Error finding container ca11fa66d7238bcfc469c55c4df22c9585e170e434ce82a525ebefdce2612516: Status 404 returned error can't find the container with id ca11fa66d7238bcfc469c55c4df22c9585e170e434ce82a525ebefdce2612516 Nov 02 20:09:39.976865 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:39.976838 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr"] Nov 02 20:09:40.403651 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:40.403611 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" event={"ID":"da593eb1-73c6-417d-ba23-e979e0866e2d","Type":"ContainerStarted","Data":"8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a"} Nov 02 20:09:40.403848 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:40.403663 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" event={"ID":"da593eb1-73c6-417d-ba23-e979e0866e2d","Type":"ContainerStarted","Data":"ca11fa66d7238bcfc469c55c4df22c9585e170e434ce82a525ebefdce2612516"} Nov 02 20:09:40.523766 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:40.523715 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" podStartSLOduration=1.523701568 podStartE2EDuration="1.523701568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:40.522072103 +0000 UTC m=+2438.864461183" watchObservedRunningTime="2026-11-02 20:09:40.523701568 +0000 UTC m=+2438.866090648" Nov 02 20:09:40.599932 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:40.599895 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:43.357440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:43.357391 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:44.762079 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:09:44.762033 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:09:44.786313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:44.786254 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:47.735002 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:47.734950 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:48.081278 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.081194 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr"] Nov 02 20:09:48.084745 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.084720 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.190929 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.190895 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a5b64d\"/\"openshift-service-ca.crt\"" Nov 02 20:09:48.191153 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.190953 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a5b64d\"/\"kube-root-ca.crt\"" Nov 02 20:09:48.191153 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.190966 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a5b64d\"/\"default-dockercfg-mbl6f\"" Nov 02 20:09:48.191153 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.191052 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a5b64d\"/\"test-model-a5b64d-kserve-self-signed-certs\"" Nov 02 20:09:48.193918 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.193890 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr"] Nov 02 20:09:48.201729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.201695 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:48.205265 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.205239 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrb2\" (UniqueName: \"kubernetes.io/projected/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-kube-api-access-zbrb2\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.205390 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.205302 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tls-certs\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.205390 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.205352 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-dshm\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.205498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.205443 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-home\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.205548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.205516 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-model-cache\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.205657 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.205636 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tmp-dir\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.306882 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.306838 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-dshm\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.307069 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.306893 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-home\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.307069 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.306951 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-model-cache\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.307069 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.307049 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tmp-dir\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.307263 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.307173 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrb2\" (UniqueName: \"kubernetes.io/projected/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-kube-api-access-zbrb2\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.307263 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.307230 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tls-certs\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.307429 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.307407 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-home\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.307516 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.307464 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tmp-dir\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.307516 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.307503 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-model-cache\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.309246 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.309222 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-dshm\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.309745 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.309723 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tls-certs\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.366005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.365928 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrb2\" (UniqueName: \"kubernetes.io/projected/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-kube-api-access-zbrb2\") pod \"test-model-a5b64d-kserve-59df48f97f-gg5kr\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.398274 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.398239 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:48.771056 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:09:48.771009 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1aeb7c5_beb6_4ef8_8701_14f2175a553b.slice/crio-cb7f6ce86f41eec2743f899813cce64ce78958c1d07dc8248fa3e0bd77ac10f9 WatchSource:0}: Error finding container cb7f6ce86f41eec2743f899813cce64ce78958c1d07dc8248fa3e0bd77ac10f9: Status 404 returned error can't find the container with id cb7f6ce86f41eec2743f899813cce64ce78958c1d07dc8248fa3e0bd77ac10f9 Nov 02 20:09:48.773128 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:48.773103 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr"] Nov 02 20:09:49.471143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:49.471096 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" event={"ID":"a1aeb7c5-beb6-4ef8-8701-14f2175a553b","Type":"ContainerStarted","Data":"95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe"} Nov 02 20:09:49.471332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:49.471151 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" event={"ID":"a1aeb7c5-beb6-4ef8-8701-14f2175a553b","Type":"ContainerStarted","Data":"cb7f6ce86f41eec2743f899813cce64ce78958c1d07dc8248fa3e0bd77ac10f9"} Nov 02 20:09:49.590457 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:49.590399 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" podStartSLOduration=1.590384059 podStartE2EDuration="1.590384059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:49.588345012 +0000 UTC m=+2447.930734093" watchObservedRunningTime="2026-11-02 20:09:49.590384059 +0000 UTC m=+2447.932773140" Nov 02 20:09:49.599818 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:49.599785 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:49.613923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:49.613896 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:50.398403 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:50.398372 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:50.489563 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:50.489532 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:09:52.122890 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:52.122843 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:56.521139 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:56.521097 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:58.398675 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:58.398632 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:58.411559 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:58.411527 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:58.549186 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:58.549153 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:09:59.143126 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:09:59.143092 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:00.946523 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:00.946487 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:05.328179 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:05.328142 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:09.678545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:09.678466 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:14.096343 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:14.096311 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:14.763025 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:10:14.762982 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:10:17.250259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:17.250226 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:18.514526 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:18.514494 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:22.842938 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:22.842902 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:27.190514 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:27.190449 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:31.699891 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:31.699849 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:32.471674 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:32.471639 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:34.419366 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:34.419316 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:34.574002 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:34.573971 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:36.233737 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:36.233700 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:36.387443 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:36.387401 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:40.944744 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:40.944706 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:43.018187 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:43.018146 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6"] Nov 02 20:10:43.018671 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:43.018357 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" containerID="cri-o://d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9" gracePeriod=30 Nov 02 20:10:44.761641 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:10:44.761601 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:10:45.354967 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:45.354929 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:48.297068 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.297000 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 20:10:48.392407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392368 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-envoy\") pod \"533d8bea-d517-4550-b754-85bedfa0200d\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " Nov 02 20:10:48.392407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392407 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533d8bea-d517-4550-b754-85bedfa0200d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533d8bea-d517-4550-b754-85bedfa0200d-istiod-ca-cert\") pod \"533d8bea-d517-4550-b754-85bedfa0200d\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " Nov 02 20:10:48.392695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392433 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-istio-token\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-istio-token\") pod \"533d8bea-d517-4550-b754-85bedfa0200d\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " Nov 02 20:10:48.392695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392519 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-data\") pod \"533d8bea-d517-4550-b754-85bedfa0200d\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " Nov 02 20:10:48.392695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392618 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-kube-api-access-nlb2h\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-kube-api-access-nlb2h\") pod \"533d8bea-d517-4550-b754-85bedfa0200d\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " Nov 02 20:10:48.392695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392672 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/533d8bea-d517-4550-b754-85bedfa0200d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533d8bea-d517-4550-b754-85bedfa0200d-istio-podinfo\") pod \"533d8bea-d517-4550-b754-85bedfa0200d\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " Nov 02 20:10:48.392930 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392697 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-credential-socket\") pod \"533d8bea-d517-4550-b754-85bedfa0200d\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " Nov 02 20:10:48.392930 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392728 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-certs\") pod \"533d8bea-d517-4550-b754-85bedfa0200d\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " Nov 02 20:10:48.392930 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392770 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-socket\") pod \"533d8bea-d517-4550-b754-85bedfa0200d\" (UID: \"533d8bea-d517-4550-b754-85bedfa0200d\") " Nov 02 20:10:48.392930 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392844 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-data" pod "533d8bea-d517-4550-b754-85bedfa0200d" (UID: "533d8bea-d517-4550-b754-85bedfa0200d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:10:48.392930 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.392890 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533d8bea-d517-4550-b754-85bedfa0200d-istiod-ca-cert" pod "533d8bea-d517-4550-b754-85bedfa0200d" (UID: "533d8bea-d517-4550-b754-85bedfa0200d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:48.393205 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.393075 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-credential-socket" pod "533d8bea-d517-4550-b754-85bedfa0200d" (UID: "533d8bea-d517-4550-b754-85bedfa0200d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:10:48.393264 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.393201 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-certs" pod "533d8bea-d517-4550-b754-85bedfa0200d" (UID: "533d8bea-d517-4550-b754-85bedfa0200d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:10:48.393317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.393286 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-socket" pod "533d8bea-d517-4550-b754-85bedfa0200d" (UID: "533d8bea-d517-4550-b754-85bedfa0200d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:10:48.393464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.393445 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533d8bea-d517-4550-b754-85bedfa0200d-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:48.393568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.393469 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:48.393568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.393509 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:48.393568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.393523 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:48.393568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.393536 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:48.394902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.394870 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/533d8bea-d517-4550-b754-85bedfa0200d-istio-podinfo" pod "533d8bea-d517-4550-b754-85bedfa0200d" (UID: "533d8bea-d517-4550-b754-85bedfa0200d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:10:48.395033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.395009 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-kube-api-access-nlb2h" pod "533d8bea-d517-4550-b754-85bedfa0200d" (UID: "533d8bea-d517-4550-b754-85bedfa0200d"). InnerVolumeSpecName "kube-api-access-nlb2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:48.395175 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.395149 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-envoy" pod "533d8bea-d517-4550-b754-85bedfa0200d" (UID: "533d8bea-d517-4550-b754-85bedfa0200d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:10:48.395256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.395242 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-istio-token" pod "533d8bea-d517-4550-b754-85bedfa0200d" (UID: "533d8bea-d517-4550-b754-85bedfa0200d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:48.494675 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.494638 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533d8bea-d517-4550-b754-85bedfa0200d-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:48.494675 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.494674 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533d8bea-d517-4550-b754-85bedfa0200d-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:48.494892 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.494685 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:48.494892 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.494694 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlb2h\" (UniqueName: \"kubernetes.io/projected/533d8bea-d517-4550-b754-85bedfa0200d-kube-api-access-nlb2h\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:48.859118 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.859085 5850 generic.go:356] "Generic (PLEG): container finished" podID="533d8bea-d517-4550-b754-85bedfa0200d" containerID="d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9" exitCode=0 Nov 02 20:10:48.859317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.859141 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" Nov 02 20:10:48.859317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.859177 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" event={"ID":"533d8bea-d517-4550-b754-85bedfa0200d","Type":"ContainerDied","Data":"d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9"} Nov 02 20:10:48.859317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.859240 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6" event={"ID":"533d8bea-d517-4550-b754-85bedfa0200d","Type":"ContainerDied","Data":"d8dc38e9b61688051f8f45c683b22db7707bcf15b96326d2b920101023f2cc1a"} Nov 02 20:10:48.859317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.859271 5850 scope.go:122] "RemoveContainer" containerID="d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9" Nov 02 20:10:48.870537 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.870512 5850 scope.go:122] "RemoveContainer" containerID="d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9" Nov 02 20:10:48.870826 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:10:48.870808 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9\": container with ID starting with d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9 not found: ID does not exist" containerID="d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9" Nov 02 20:10:48.870902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:48.870834 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9"} err="failed to get container status \"d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9\": rpc error: code = NotFound desc = could not find container \"d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9\": container with ID starting with d482b85386ab05dd89aaab870df0eeaff6f83a4d7fc7371fbec7e8eaa35d33e9 not found: ID does not exist" Nov 02 20:10:49.010672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:49.010634 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4"] Nov 02 20:10:49.029114 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:49.029066 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6"] Nov 02 20:10:49.062312 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:49.062273 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-nqgj6"] Nov 02 20:10:49.125205 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:49.125169 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn"] Nov 02 20:10:49.411028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:49.410929 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:49.574256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:49.574023 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:49.737003 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:49.736902 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:49.815906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:49.815872 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533d8bea-d517-4550-b754-85bedfa0200d" path="/var/lib/kubelet/pods/533d8bea-d517-4550-b754-85bedfa0200d/volumes" Nov 02 20:10:54.011097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.011051 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" podUID="5c1c21e9-8348-4d6d-bff8-fda75236b9e3" containerName="payload-processing" containerID="cri-o://57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662" gracePeriod=25 Nov 02 20:10:54.103744 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.103698 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:54.126823 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.126595 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" podUID="1b6821f8-53b3-4ec3-b983-66e93eda9e86" containerName="payload-pre-processing" containerID="cri-o://22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168" gracePeriod=25 Nov 02 20:10:54.329624 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.329591 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:10:54.373858 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.373825 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-kube-api-access-vv6qw\") pod \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\" (UID: \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\") " Nov 02 20:10:54.374066 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.374030 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-plugins-config-volume\") pod \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\" (UID: \"5c1c21e9-8348-4d6d-bff8-fda75236b9e3\") " Nov 02 20:10:54.374532 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.374490 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-plugins-config-volume" pod "5c1c21e9-8348-4d6d-bff8-fda75236b9e3" (UID: "5c1c21e9-8348-4d6d-bff8-fda75236b9e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:54.376731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.376704 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-kube-api-access-vv6qw" pod "5c1c21e9-8348-4d6d-bff8-fda75236b9e3" (UID: "5c1c21e9-8348-4d6d-bff8-fda75236b9e3"). InnerVolumeSpecName "kube-api-access-vv6qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:54.395654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.395628 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:10:54.475682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.475639 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6821f8-53b3-4ec3-b983-66e93eda9e86-kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/1b6821f8-53b3-4ec3-b983-66e93eda9e86-kube-api-access-f5lxl\") pod \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\" (UID: \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\") " Nov 02 20:10:54.475895 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.475712 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6821f8-53b3-4ec3-b983-66e93eda9e86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6821f8-53b3-4ec3-b983-66e93eda9e86-plugins-config-volume\") pod \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\" (UID: \"1b6821f8-53b3-4ec3-b983-66e93eda9e86\") " Nov 02 20:10:54.476171 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.476140 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6821f8-53b3-4ec3-b983-66e93eda9e86-plugins-config-volume" pod "1b6821f8-53b3-4ec3-b983-66e93eda9e86" (UID: "1b6821f8-53b3-4ec3-b983-66e93eda9e86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:54.476317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.476299 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-kube-api-access-vv6qw\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:54.476395 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.476318 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6821f8-53b3-4ec3-b983-66e93eda9e86-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:54.476395 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.476328 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c21e9-8348-4d6d-bff8-fda75236b9e3-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:54.478586 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.478548 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6821f8-53b3-4ec3-b983-66e93eda9e86-kube-api-access-f5lxl" pod "1b6821f8-53b3-4ec3-b983-66e93eda9e86" (UID: "1b6821f8-53b3-4ec3-b983-66e93eda9e86"). InnerVolumeSpecName "kube-api-access-f5lxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:54.577971 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.577880 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5lxl\" (UniqueName: \"kubernetes.io/projected/1b6821f8-53b3-4ec3-b983-66e93eda9e86-kube-api-access-f5lxl\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:54.915951 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.915917 5850 generic.go:356] "Generic (PLEG): container finished" podID="1b6821f8-53b3-4ec3-b983-66e93eda9e86" containerID="22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168" exitCode=0 Nov 02 20:10:54.916142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.915980 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" Nov 02 20:10:54.916142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.915997 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" event={"ID":"1b6821f8-53b3-4ec3-b983-66e93eda9e86","Type":"ContainerDied","Data":"22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168"} Nov 02 20:10:54.916142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.916077 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn" event={"ID":"1b6821f8-53b3-4ec3-b983-66e93eda9e86","Type":"ContainerDied","Data":"507039f289bf306fea013bd68ebe1ac9b33dce9b9bc1e8b4515f17bebe2ea97d"} Nov 02 20:10:54.916142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.916107 5850 scope.go:122] "RemoveContainer" containerID="22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168" Nov 02 20:10:54.917668 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.917641 5850 generic.go:356] "Generic (PLEG): container finished" podID="5c1c21e9-8348-4d6d-bff8-fda75236b9e3" containerID="57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662" exitCode=0 Nov 02 20:10:54.917808 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.917692 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" Nov 02 20:10:54.917808 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.917751 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" event={"ID":"5c1c21e9-8348-4d6d-bff8-fda75236b9e3","Type":"ContainerDied","Data":"57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662"} Nov 02 20:10:54.917808 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.917791 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4" event={"ID":"5c1c21e9-8348-4d6d-bff8-fda75236b9e3","Type":"ContainerDied","Data":"3843c4250292d75fb193cca99dc2a11c2c846685c2ae4be3a40a095fb94b50d3"} Nov 02 20:10:54.926858 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.926840 5850 scope.go:122] "RemoveContainer" containerID="22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168" Nov 02 20:10:54.927152 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:10:54.927130 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168\": container with ID starting with 22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168 not found: ID does not exist" containerID="22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168" Nov 02 20:10:54.927266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.927157 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168"} err="failed to get container status \"22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168\": rpc error: code = NotFound desc = could not find container \"22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168\": container with ID starting with 22f82ae73e8f501313d3e5a45289f50e9d2598bd0648c60ce81ff19c3ce72168 not found: ID does not exist" Nov 02 20:10:54.927266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.927175 5850 scope.go:122] "RemoveContainer" containerID="57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662" Nov 02 20:10:54.936404 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.936385 5850 scope.go:122] "RemoveContainer" containerID="57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662" Nov 02 20:10:54.936717 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:10:54.936698 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662\": container with ID starting with 57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662 not found: ID does not exist" containerID="57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662" Nov 02 20:10:54.936768 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:54.936725 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662"} err="failed to get container status \"57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662\": rpc error: code = NotFound desc = could not find container \"57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662\": container with ID starting with 57a7a6afe06766951ab93e5f1ba53b45918e40ed445457edacf26dc461510662 not found: ID does not exist" Nov 02 20:10:55.086137 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:55.086098 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4"] Nov 02 20:10:55.088491 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:55.088455 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1db0e4-7486f748d7-blwn4"] Nov 02 20:10:55.251669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:55.251580 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn"] Nov 02 20:10:55.254142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:55.254110 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1db0e4-7f758d5c7f-5prbn"] Nov 02 20:10:55.816023 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:55.815979 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6821f8-53b3-4ec3-b983-66e93eda9e86" path="/var/lib/kubelet/pods/1b6821f8-53b3-4ec3-b983-66e93eda9e86/volumes" Nov 02 20:10:55.816561 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:55.816540 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1c21e9-8348-4d6d-bff8-fda75236b9e3" path="/var/lib/kubelet/pods/5c1c21e9-8348-4d6d-bff8-fda75236b9e3/volumes" Nov 02 20:10:58.481983 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:10:58.481928 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:02.830580 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:02.830540 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:07.239667 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:07.239616 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:11.700190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:11.700080 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:14.763488 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:11:14.763433 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:11:16.048940 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:16.048904 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:17.558488 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:17.558455 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:20.432367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:20.432317 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:24.821247 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:24.821200 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:29.286753 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:29.286715 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:33.582682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:33.582640 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:37.975493 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:37.975429 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:42.275588 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:42.275550 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:43.301547 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:43.301511 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q"] Nov 02 20:11:43.302019 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:43.301774 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" containerID="cri-o://99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e" gracePeriod=30 Nov 02 20:11:44.763291 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:11:44.763247 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:11:45.005289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:45.005242 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:11:45.005501 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:45.005312 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:11:46.632687 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:46.632652 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:48.563869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.563841 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 20:11:48.617097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617060 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-kube-api-access-hm6gd\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-kube-api-access-hm6gd\") pod \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " Nov 02 20:11:48.617454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617125 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-envoy\") pod \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " Nov 02 20:11:48.617454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617154 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-token\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-token\") pod \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " Nov 02 20:11:48.617454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617216 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-certs\") pod \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " Nov 02 20:11:48.617454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617262 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-credential-socket\") pod \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " Nov 02 20:11:48.617454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617324 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-socket\") pod \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " Nov 02 20:11:48.617454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617378 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-podinfo\") pod \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " Nov 02 20:11:48.617862 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617500 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-data\") pod \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " Nov 02 20:11:48.617862 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617636 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-socket" pod "f3dc06fe-3a71-49a9-9cdd-98a54700ca96" (UID: "f3dc06fe-3a71-49a9-9cdd-98a54700ca96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:48.617862 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617652 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-certs" pod "f3dc06fe-3a71-49a9-9cdd-98a54700ca96" (UID: "f3dc06fe-3a71-49a9-9cdd-98a54700ca96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:48.618035 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617880 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-data" pod "f3dc06fe-3a71-49a9-9cdd-98a54700ca96" (UID: "f3dc06fe-3a71-49a9-9cdd-98a54700ca96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:48.618035 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.617944 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-credential-socket" pod "f3dc06fe-3a71-49a9-9cdd-98a54700ca96" (UID: "f3dc06fe-3a71-49a9-9cdd-98a54700ca96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:48.618143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.618078 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:48.618143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.618094 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:48.618143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.618109 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:48.618143 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.618123 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:48.619660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.619636 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-kube-api-access-hm6gd" pod "f3dc06fe-3a71-49a9-9cdd-98a54700ca96" (UID: "f3dc06fe-3a71-49a9-9cdd-98a54700ca96"). InnerVolumeSpecName "kube-api-access-hm6gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:11:48.619781 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.619728 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-podinfo" pod "f3dc06fe-3a71-49a9-9cdd-98a54700ca96" (UID: "f3dc06fe-3a71-49a9-9cdd-98a54700ca96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:11:48.619895 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.619879 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-token" pod "f3dc06fe-3a71-49a9-9cdd-98a54700ca96" (UID: "f3dc06fe-3a71-49a9-9cdd-98a54700ca96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:11:48.619958 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.619928 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-envoy" pod "f3dc06fe-3a71-49a9-9cdd-98a54700ca96" (UID: "f3dc06fe-3a71-49a9-9cdd-98a54700ca96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:48.719611 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.719574 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istiod-ca-cert\") pod \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\" (UID: \"f3dc06fe-3a71-49a9-9cdd-98a54700ca96\") " Nov 02 20:11:48.720055 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.720028 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istiod-ca-cert" pod "f3dc06fe-3a71-49a9-9cdd-98a54700ca96" (UID: "f3dc06fe-3a71-49a9-9cdd-98a54700ca96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:11:48.720214 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.720199 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:48.720260 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.720217 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:48.720260 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.720230 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6gd\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-kube-api-access-hm6gd\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:48.720260 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.720241 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:48.720260 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:48.720250 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3dc06fe-3a71-49a9-9cdd-98a54700ca96-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:49.256825 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.256790 5850 generic.go:356] "Generic (PLEG): container finished" podID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerID="99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e" exitCode=0 Nov 02 20:11:49.257019 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.256838 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" event={"ID":"f3dc06fe-3a71-49a9-9cdd-98a54700ca96","Type":"ContainerDied","Data":"99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e"} Nov 02 20:11:49.257019 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.256883 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" event={"ID":"f3dc06fe-3a71-49a9-9cdd-98a54700ca96","Type":"ContainerDied","Data":"968bed8fa9f7c4adfe9aef52149ca6ff5d174979de2dca563a040da89a175e67"} Nov 02 20:11:49.257019 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.256905 5850 scope.go:122] "RemoveContainer" containerID="99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e" Nov 02 20:11:49.257019 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.256842 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q" Nov 02 20:11:49.267593 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.267572 5850 scope.go:122] "RemoveContainer" containerID="99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e" Nov 02 20:11:49.267919 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:11:49.267896 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e\": container with ID starting with 99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e not found: ID does not exist" containerID="99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e" Nov 02 20:11:49.267995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.267927 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e"} err="failed to get container status \"99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e\": rpc error: code = NotFound desc = could not find container \"99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e\": container with ID starting with 99dc7936bf307c8e5f9d0747716bcde24e784ef1e1a5c69e1247e65e98b0115e not found: ID does not exist" Nov 02 20:11:49.427087 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.427048 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q"] Nov 02 20:11:49.478798 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.478760 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jrr5q"] Nov 02 20:11:49.816169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:49.816136 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" path="/var/lib/kubelet/pods/f3dc06fe-3a71-49a9-9cdd-98a54700ca96/volumes" Nov 02 20:11:51.024915 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:51.024863 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:55.451859 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:55.451825 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:59.846032 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:11:59.845990 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:04.293989 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:04.293948 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:08.623248 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:08.623216 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:13.036268 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:13.036224 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:14.762072 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:12:14.762028 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:12:17.411403 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:17.411362 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:17.879098 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:17.879061 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:18.340766 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:18.340649 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:21.802466 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:21.802417 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:26.124816 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:26.124779 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:30.468265 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:30.468225 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:34.391135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:34.391096 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:35.878147 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:35.878109 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:36.123689 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:36.123648 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:36.277959 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:36.277872 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:36.822949 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:36.822912 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:40.090137 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:40.090058 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:40.785999 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:40.785961 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:42.478803 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:42.478765 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:42.638787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:42.638754 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:42.771536 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:42.771437 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7"] Nov 02 20:12:42.771744 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:42.771701 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" containerID="cri-o://a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce" gracePeriod=30 Nov 02 20:12:42.794709 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:42.794671 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:44.762282 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:12:44.762243 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:12:45.255819 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:45.255774 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:48.037502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.037406 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:12:48.135296 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135260 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-podinfo\") pod \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " Nov 02 20:12:48.135506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135313 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istiod-ca-cert\") pod \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " Nov 02 20:12:48.135506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135419 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-data\") pod \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " Nov 02 20:12:48.135506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135456 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-certs\") pod \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " Nov 02 20:12:48.135717 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135507 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-credential-socket\") pod \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " Nov 02 20:12:48.135717 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135538 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-token\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-token\") pod \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " Nov 02 20:12:48.135717 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135560 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-envoy\") pod \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " Nov 02 20:12:48.135717 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135605 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-kube-api-access-b25xc\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-kube-api-access-b25xc\") pod \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " Nov 02 20:12:48.135911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135738 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-socket\") pod \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\" (UID: \"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0\") " Nov 02 20:12:48.135911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135747 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-data" pod "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" (UID: "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:12:48.135911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135779 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-certs" pod "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" (UID: "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:12:48.135911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.135794 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istiod-ca-cert" pod "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" (UID: "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:12:48.136126 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.136074 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-socket" pod "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" (UID: "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:12:48.136230 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.136202 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-credential-socket" pod "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" (UID: "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:12:48.136386 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.136367 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:48.136465 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.136391 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:48.136465 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.136408 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:48.136465 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.136422 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:48.136465 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.136436 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:48.138080 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.138053 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-kube-api-access-b25xc" pod "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" (UID: "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0"). InnerVolumeSpecName "kube-api-access-b25xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:12:48.138194 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.138086 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-token" pod "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" (UID: "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:12:48.138194 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.138127 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-envoy" pod "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" (UID: "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:12:48.138329 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.138306 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-podinfo" pod "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" (UID: "91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:12:48.237957 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.237914 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b25xc\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-kube-api-access-b25xc\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:48.237957 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.237949 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:48.237957 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.237959 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:48.237957 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.237968 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:48.657913 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.657872 5850 generic.go:356] "Generic (PLEG): container finished" podID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerID="a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce" exitCode=0 Nov 02 20:12:48.658104 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.657927 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" Nov 02 20:12:48.658104 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.657950 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" event={"ID":"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0","Type":"ContainerDied","Data":"a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce"} Nov 02 20:12:48.658104 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.658002 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7" event={"ID":"91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0","Type":"ContainerDied","Data":"a3c4691486f7b7533f630c2eb5d8a4bbc9124c915a0669479480af8a96c3e277"} Nov 02 20:12:48.658104 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.658029 5850 scope.go:122] "RemoveContainer" containerID="a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce" Nov 02 20:12:48.669088 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.669067 5850 scope.go:122] "RemoveContainer" containerID="256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317" Nov 02 20:12:48.678308 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.678286 5850 scope.go:122] "RemoveContainer" containerID="a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce" Nov 02 20:12:48.678657 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:12:48.678633 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce\": container with ID starting with a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce not found: ID does not exist" containerID="a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce" Nov 02 20:12:48.678714 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.678671 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce"} err="failed to get container status \"a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce\": rpc error: code = NotFound desc = could not find container \"a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce\": container with ID starting with a6c15505c03c8da04f7fbb3e151372181d7865299213e444df4a68ee505967ce not found: ID does not exist" Nov 02 20:12:48.678714 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.678690 5850 scope.go:122] "RemoveContainer" containerID="256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317" Nov 02 20:12:48.678958 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:12:48.678937 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317\": container with ID starting with 256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317 not found: ID does not exist" containerID="256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317" Nov 02 20:12:48.679020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.678970 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317"} err="failed to get container status \"256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317\": rpc error: code = NotFound desc = could not find container \"256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317\": container with ID starting with 256e6b1a7eb35ed70cdfd08b1be16fb35db8156d0941dad8e1dc7bce9388c317 not found: ID does not exist" Nov 02 20:12:48.827383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.827346 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7"] Nov 02 20:12:48.832305 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:48.832275 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1db0e4-openshift-default-579489bd77-xkqk7"] Nov 02 20:12:49.817041 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:49.817005 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" path="/var/lib/kubelet/pods/91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0/volumes" Nov 02 20:12:49.958667 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:49.958637 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:54.282600 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:54.282562 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:55.527840 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:55.527795 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:55.689092 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:55.689043 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:58.645620 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:12:58.645579 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:03.190500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:03.189771 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:07.620923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:07.620887 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:11.956436 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:11.956397 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:14.762908 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:13:14.762868 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:13:16.347716 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:16.347676 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:18.200923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:18.200892 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:18.651984 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:18.651941 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:20.854919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:20.854855 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:25.235634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:25.235598 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:29.297196 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:29.297157 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr"] Nov 02 20:13:29.297643 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:29.297384 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" containerID="cri-o://77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f" gracePeriod=30 Nov 02 20:13:29.676672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:29.676634 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:29.731977 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:29.731938 5850 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:13:29.732282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:29.732254 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:13:30.611068 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:30.611027 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:34.011395 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.011360 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:34.565545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.565452 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 20:13:34.686331 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686291 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-kube-api-access-b79v8\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-kube-api-access-b79v8\") pod \"ad146ede-1b0e-4d32-b7bd-c95590295f34\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " Nov 02 20:13:34.686579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686412 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad146ede-1b0e-4d32-b7bd-c95590295f34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad146ede-1b0e-4d32-b7bd-c95590295f34-istiod-ca-cert\") pod \"ad146ede-1b0e-4d32-b7bd-c95590295f34\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " Nov 02 20:13:34.686579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686448 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-envoy\") pod \"ad146ede-1b0e-4d32-b7bd-c95590295f34\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " Nov 02 20:13:34.686579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686465 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-token\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-token\") pod \"ad146ede-1b0e-4d32-b7bd-c95590295f34\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " Nov 02 20:13:34.686579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686520 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-socket\") pod \"ad146ede-1b0e-4d32-b7bd-c95590295f34\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " Nov 02 20:13:34.686579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686549 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-credential-socket\") pod \"ad146ede-1b0e-4d32-b7bd-c95590295f34\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " Nov 02 20:13:34.686874 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686652 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-data\") pod \"ad146ede-1b0e-4d32-b7bd-c95590295f34\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " Nov 02 20:13:34.686874 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686694 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-certs\") pod \"ad146ede-1b0e-4d32-b7bd-c95590295f34\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " Nov 02 20:13:34.686874 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686796 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-podinfo\") pod \"ad146ede-1b0e-4d32-b7bd-c95590295f34\" (UID: \"ad146ede-1b0e-4d32-b7bd-c95590295f34\") " Nov 02 20:13:34.687038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686880 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-socket" pod "ad146ede-1b0e-4d32-b7bd-c95590295f34" (UID: "ad146ede-1b0e-4d32-b7bd-c95590295f34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:34.687038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686899 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-credential-socket" pod "ad146ede-1b0e-4d32-b7bd-c95590295f34" (UID: "ad146ede-1b0e-4d32-b7bd-c95590295f34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:34.687038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.687019 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-data" pod "ad146ede-1b0e-4d32-b7bd-c95590295f34" (UID: "ad146ede-1b0e-4d32-b7bd-c95590295f34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:34.687038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.686945 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad146ede-1b0e-4d32-b7bd-c95590295f34-istiod-ca-cert" pod "ad146ede-1b0e-4d32-b7bd-c95590295f34" (UID: "ad146ede-1b0e-4d32-b7bd-c95590295f34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:13:34.687258 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.687116 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-certs" pod "ad146ede-1b0e-4d32-b7bd-c95590295f34" (UID: "ad146ede-1b0e-4d32-b7bd-c95590295f34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:34.687455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.687430 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:34.687570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.687461 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:34.687570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.687502 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad146ede-1b0e-4d32-b7bd-c95590295f34-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:34.687570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.687518 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:34.687570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.687534 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:34.688944 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.688917 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-token" pod "ad146ede-1b0e-4d32-b7bd-c95590295f34" (UID: "ad146ede-1b0e-4d32-b7bd-c95590295f34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:13:34.689058 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.688947 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-podinfo" pod "ad146ede-1b0e-4d32-b7bd-c95590295f34" (UID: "ad146ede-1b0e-4d32-b7bd-c95590295f34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:13:34.689058 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.688954 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-envoy" pod "ad146ede-1b0e-4d32-b7bd-c95590295f34" (UID: "ad146ede-1b0e-4d32-b7bd-c95590295f34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:34.689058 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.689033 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-kube-api-access-b79v8" pod "ad146ede-1b0e-4d32-b7bd-c95590295f34" (UID: "ad146ede-1b0e-4d32-b7bd-c95590295f34"). InnerVolumeSpecName "kube-api-access-b79v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:13:34.788739 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.788699 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:34.788739 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.788741 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:34.788968 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.788756 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad146ede-1b0e-4d32-b7bd-c95590295f34-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:34.788968 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.788771 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b79v8\" (UniqueName: \"kubernetes.io/projected/ad146ede-1b0e-4d32-b7bd-c95590295f34-kube-api-access-b79v8\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:34.985095 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.985060 5850 generic.go:356] "Generic (PLEG): container finished" podID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerID="77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f" exitCode=0 Nov 02 20:13:34.985256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.985110 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" Nov 02 20:13:34.985256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.985141 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" event={"ID":"ad146ede-1b0e-4d32-b7bd-c95590295f34","Type":"ContainerDied","Data":"77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f"} Nov 02 20:13:34.985256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.985203 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr" event={"ID":"ad146ede-1b0e-4d32-b7bd-c95590295f34","Type":"ContainerDied","Data":"73b22f87a564ab0a48aeba4ea7c1c8327fce050a7b054f6c7e9ffdba355ffa26"} Nov 02 20:13:34.985256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.985228 5850 scope.go:122] "RemoveContainer" containerID="77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f" Nov 02 20:13:34.995687 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.995669 5850 scope.go:122] "RemoveContainer" containerID="77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f" Nov 02 20:13:34.996009 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:13:34.995990 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f\": container with ID starting with 77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f not found: ID does not exist" containerID="77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f" Nov 02 20:13:34.996062 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:34.996017 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f"} err="failed to get container status \"77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f\": rpc error: code = NotFound desc = could not find container \"77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f\": container with ID starting with 77f48025780286dc6bac86b35cb08aaa97b4226677434c6db135ba36f032651f not found: ID does not exist" Nov 02 20:13:35.155828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:35.155794 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr"] Nov 02 20:13:35.207363 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:35.207326 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-wr4lr"] Nov 02 20:13:35.815774 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:35.815744 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" path="/var/lib/kubelet/pods/ad146ede-1b0e-4d32-b7bd-c95590295f34/volumes" Nov 02 20:13:38.342000 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:38.341966 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:40.556458 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:40.556419 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:40.714101 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:40.714061 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:42.731223 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:42.731184 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:44.761909 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:13:44.761859 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:13:47.104127 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:47.104094 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:51.473531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:51.473465 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:52.416660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:52.416629 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:52.820404 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:52.820323 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:53.471241 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:53.471194 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:55.813768 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:13:55.813735 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:00.199835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:00.199791 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:04.552752 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:04.552716 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:08.873290 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:08.873257 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:13.245163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:13.245132 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:14.387748 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:14.387705 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j"] Nov 02 20:14:14.388257 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:14.387922 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" containerID="cri-o://e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8" gracePeriod=30 Nov 02 20:14:14.762824 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:14:14.762735 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:14:15.056614 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:15.056520 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:14:15.056614 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:15.056589 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:14:17.767680 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:17.767625 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:18.532487 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:18.532439 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:19.666930 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.666865 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 20:14:19.789137 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789099 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-socket\") pod \"2c1ee935-9e4f-4e64-8e87-7821569346cc\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " Nov 02 20:14:19.789319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789161 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-data\") pod \"2c1ee935-9e4f-4e64-8e87-7821569346cc\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " Nov 02 20:14:19.789319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789230 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-kube-api-access-nwsv7\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-kube-api-access-nwsv7\") pod \"2c1ee935-9e4f-4e64-8e87-7821569346cc\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " Nov 02 20:14:19.789319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789271 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1ee935-9e4f-4e64-8e87-7821569346cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1ee935-9e4f-4e64-8e87-7821569346cc-istiod-ca-cert\") pod \"2c1ee935-9e4f-4e64-8e87-7821569346cc\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " Nov 02 20:14:19.789460 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789326 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-podinfo\") pod \"2c1ee935-9e4f-4e64-8e87-7821569346cc\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " Nov 02 20:14:19.789460 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789357 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-credential-socket\") pod \"2c1ee935-9e4f-4e64-8e87-7821569346cc\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " Nov 02 20:14:19.789460 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789397 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-envoy\") pod \"2c1ee935-9e4f-4e64-8e87-7821569346cc\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " Nov 02 20:14:19.789460 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789437 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-certs\") pod \"2c1ee935-9e4f-4e64-8e87-7821569346cc\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " Nov 02 20:14:19.789460 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789435 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-socket" pod "2c1ee935-9e4f-4e64-8e87-7821569346cc" (UID: "2c1ee935-9e4f-4e64-8e87-7821569346cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.789460 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789451 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-data" pod "2c1ee935-9e4f-4e64-8e87-7821569346cc" (UID: "2c1ee935-9e4f-4e64-8e87-7821569346cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.789772 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789593 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-token\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-token\") pod \"2c1ee935-9e4f-4e64-8e87-7821569346cc\" (UID: \"2c1ee935-9e4f-4e64-8e87-7821569346cc\") " Nov 02 20:14:19.789857 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789793 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-certs" pod "2c1ee935-9e4f-4e64-8e87-7821569346cc" (UID: "2c1ee935-9e4f-4e64-8e87-7821569346cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.789857 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789844 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-credential-socket" pod "2c1ee935-9e4f-4e64-8e87-7821569346cc" (UID: "2c1ee935-9e4f-4e64-8e87-7821569346cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.789969 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.789853 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1ee935-9e4f-4e64-8e87-7821569346cc-istiod-ca-cert" pod "2c1ee935-9e4f-4e64-8e87-7821569346cc" (UID: "2c1ee935-9e4f-4e64-8e87-7821569346cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:19.790278 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.790253 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1ee935-9e4f-4e64-8e87-7821569346cc-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.790278 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.790275 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.790489 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.790287 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.790489 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.790305 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.790489 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.790320 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.791790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.791763 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-kube-api-access-nwsv7" pod "2c1ee935-9e4f-4e64-8e87-7821569346cc" (UID: "2c1ee935-9e4f-4e64-8e87-7821569346cc"). InnerVolumeSpecName "kube-api-access-nwsv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:19.791908 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.791850 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-token" pod "2c1ee935-9e4f-4e64-8e87-7821569346cc" (UID: "2c1ee935-9e4f-4e64-8e87-7821569346cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:19.792073 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.792057 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-envoy" pod "2c1ee935-9e4f-4e64-8e87-7821569346cc" (UID: "2c1ee935-9e4f-4e64-8e87-7821569346cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.792148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.792126 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-podinfo" pod "2c1ee935-9e4f-4e64-8e87-7821569346cc" (UID: "2c1ee935-9e4f-4e64-8e87-7821569346cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:14:19.892168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.892138 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.892168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.892165 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwsv7\" (UniqueName: \"kubernetes.io/projected/2c1ee935-9e4f-4e64-8e87-7821569346cc-kube-api-access-nwsv7\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.892168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.892178 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.892406 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:19.892188 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1ee935-9e4f-4e64-8e87-7821569346cc-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:20.262066 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:20.262030 5850 generic.go:356] "Generic (PLEG): container finished" podID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerID="e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8" exitCode=0 Nov 02 20:14:20.262281 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:20.262079 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" Nov 02 20:14:20.262281 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:20.262116 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" event={"ID":"2c1ee935-9e4f-4e64-8e87-7821569346cc","Type":"ContainerDied","Data":"e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8"} Nov 02 20:14:20.262281 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:20.262175 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j" event={"ID":"2c1ee935-9e4f-4e64-8e87-7821569346cc","Type":"ContainerDied","Data":"56e71f342a2ecc78660e48e9d242e83671065e6f6ec41118afe5868a9c4275d0"} Nov 02 20:14:20.262281 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:20.262197 5850 scope.go:122] "RemoveContainer" containerID="e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8" Nov 02 20:14:20.272347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:20.272322 5850 scope.go:122] "RemoveContainer" containerID="e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8" Nov 02 20:14:20.272694 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:14:20.272671 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8\": container with ID starting with e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8 not found: ID does not exist" containerID="e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8" Nov 02 20:14:20.272784 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:20.272703 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8"} err="failed to get container status \"e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8\": rpc error: code = NotFound desc = could not find container \"e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8\": container with ID starting with e0f2a221b8e49b46ed354afa7d1940a9ef32574612c65415b50aa4861963b6f8 not found: ID does not exist" Nov 02 20:14:20.437968 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:20.437929 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j"] Nov 02 20:14:20.490846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:20.490809 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2xg2j"] Nov 02 20:14:21.346879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:21.346837 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:21.816990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:21.816950 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" path="/var/lib/kubelet/pods/2c1ee935-9e4f-4e64-8e87-7821569346cc/volumes" Nov 02 20:14:22.220527 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:22.220497 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:26.852752 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:26.852711 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:31.230874 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:31.230837 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:33.860950 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:33.860910 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:34.111619 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:34.111537 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:34.944324 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:34.944284 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr"] Nov 02 20:14:34.980956 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:34.980914 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:35.618499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:35.618445 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:39.928756 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:39.928724 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:44.336955 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:44.336920 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:44.762783 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:14:44.762747 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:14:48.744587 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:48.744549 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:49.965151 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:49.965100 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" podUID="da593eb1-73c6-417d-ba23-e979e0866e2d" containerName="main" containerID="cri-o://8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a" gracePeriod=45 Nov 02 20:14:50.229953 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.229872 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:14:50.301068 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301030 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da593eb1-73c6-417d-ba23-e979e0866e2d-kube-api-access-rpkgc\" (UniqueName: \"kubernetes.io/projected/da593eb1-73c6-417d-ba23-e979e0866e2d-kube-api-access-rpkgc\") pod \"da593eb1-73c6-417d-ba23-e979e0866e2d\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " Nov 02 20:14:50.301255 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301153 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da593eb1-73c6-417d-ba23-e979e0866e2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/da593eb1-73c6-417d-ba23-e979e0866e2d-tls-certs\") pod \"da593eb1-73c6-417d-ba23-e979e0866e2d\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " Nov 02 20:14:50.301298 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301255 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-model-cache\") pod \"da593eb1-73c6-417d-ba23-e979e0866e2d\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " Nov 02 20:14:50.301298 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301294 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-tmp-dir\") pod \"da593eb1-73c6-417d-ba23-e979e0866e2d\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " Nov 02 20:14:50.301403 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301318 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-dshm\") pod \"da593eb1-73c6-417d-ba23-e979e0866e2d\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " Nov 02 20:14:50.301403 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301376 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-home\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-home\") pod \"da593eb1-73c6-417d-ba23-e979e0866e2d\" (UID: \"da593eb1-73c6-417d-ba23-e979e0866e2d\") " Nov 02 20:14:50.301650 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301616 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-model-cache" pod "da593eb1-73c6-417d-ba23-e979e0866e2d" (UID: "da593eb1-73c6-417d-ba23-e979e0866e2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:50.301650 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301633 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-tmp-dir" pod "da593eb1-73c6-417d-ba23-e979e0866e2d" (UID: "da593eb1-73c6-417d-ba23-e979e0866e2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:50.301791 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301735 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-home" pod "da593eb1-73c6-417d-ba23-e979e0866e2d" (UID: "da593eb1-73c6-417d-ba23-e979e0866e2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:50.301927 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301908 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:50.301987 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301932 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:50.301987 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.301946 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:50.303928 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.303878 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-dshm" pod "da593eb1-73c6-417d-ba23-e979e0866e2d" (UID: "da593eb1-73c6-417d-ba23-e979e0866e2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:50.304021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.303922 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da593eb1-73c6-417d-ba23-e979e0866e2d-kube-api-access-rpkgc" pod "da593eb1-73c6-417d-ba23-e979e0866e2d" (UID: "da593eb1-73c6-417d-ba23-e979e0866e2d"). InnerVolumeSpecName "kube-api-access-rpkgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:50.304021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.303909 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da593eb1-73c6-417d-ba23-e979e0866e2d-tls-certs" pod "da593eb1-73c6-417d-ba23-e979e0866e2d" (UID: "da593eb1-73c6-417d-ba23-e979e0866e2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:50.403461 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.403422 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da593eb1-73c6-417d-ba23-e979e0866e2d-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:50.403461 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.403457 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da593eb1-73c6-417d-ba23-e979e0866e2d-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:50.403708 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.403468 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpkgc\" (UniqueName: \"kubernetes.io/projected/da593eb1-73c6-417d-ba23-e979e0866e2d-kube-api-access-rpkgc\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:50.466952 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.466914 5850 generic.go:356] "Generic (PLEG): container finished" podID="da593eb1-73c6-417d-ba23-e979e0866e2d" containerID="8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a" exitCode=0 Nov 02 20:14:50.467155 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.466962 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" Nov 02 20:14:50.467155 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.466999 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" event={"ID":"da593eb1-73c6-417d-ba23-e979e0866e2d","Type":"ContainerDied","Data":"8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a"} Nov 02 20:14:50.467155 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.467054 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr" event={"ID":"da593eb1-73c6-417d-ba23-e979e0866e2d","Type":"ContainerDied","Data":"ca11fa66d7238bcfc469c55c4df22c9585e170e434ce82a525ebefdce2612516"} Nov 02 20:14:50.467155 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.467077 5850 scope.go:122] "RemoveContainer" containerID="8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a" Nov 02 20:14:50.477741 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.477722 5850 scope.go:122] "RemoveContainer" containerID="8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a" Nov 02 20:14:50.478035 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:14:50.478015 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a\": container with ID starting with 8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a not found: ID does not exist" containerID="8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a" Nov 02 20:14:50.478109 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.478043 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a"} err="failed to get container status \"8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a\": rpc error: code = NotFound desc = could not find container \"8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a\": container with ID starting with 8dc9adc6789fd7638d0386e2e375a71fb577fdc2802c3665cce0c3aa1205260a not found: ID does not exist" Nov 02 20:14:50.650498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.649540 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr"] Nov 02 20:14:50.650498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:50.649603 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7025fa/test-model-7025fa-kserve-d6f649cc-bccrr"] Nov 02 20:14:51.817600 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:51.817568 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da593eb1-73c6-417d-ba23-e979e0866e2d" path="/var/lib/kubelet/pods/da593eb1-73c6-417d-ba23-e979e0866e2d/volumes" Nov 02 20:14:53.220188 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:53.220154 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:57.320829 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:57.320790 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:57.562390 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:14:57.562353 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:00.189176 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189131 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7"] Nov 02 20:15:00.189843 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189815 5850 cpu_manager.go:414] "containerMap: removing container" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" Nov 02 20:15:00.189843 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189840 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189858 5850 cpu_manager.go:414] "containerMap: removing container" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189864 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189880 5850 cpu_manager.go:414] "containerMap: removing container" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189885 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189899 5850 cpu_manager.go:414] "containerMap: removing container" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189904 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189916 5850 cpu_manager.go:414] "containerMap: removing container" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189921 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189927 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1b6821f8-53b3-4ec3-b983-66e93eda9e86" containerName="payload-pre-processing" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189931 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6821f8-53b3-4ec3-b983-66e93eda9e86" containerName="payload-pre-processing" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189946 5850 cpu_manager.go:414] "containerMap: removing container" podUID="da593eb1-73c6-417d-ba23-e979e0866e2d" containerName="main" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189950 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da593eb1-73c6-417d-ba23-e979e0866e2d" containerName="main" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189957 5850 cpu_manager.go:414] "containerMap: removing container" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189962 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189969 5850 cpu_manager.go:414] "containerMap: removing container" podUID="5c1c21e9-8348-4d6d-bff8-fda75236b9e3" containerName="payload-processing" Nov 02 20:15:00.190028 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.189973 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1c21e9-8348-4d6d-bff8-fda75236b9e3" containerName="payload-processing" Nov 02 20:15:00.190616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.190056 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3dc06fe-3a71-49a9-9cdd-98a54700ca96" containerName="istio-proxy" Nov 02 20:15:00.190616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.190064 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6821f8-53b3-4ec3-b983-66e93eda9e86" containerName="payload-pre-processing" Nov 02 20:15:00.190616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.190074 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="da593eb1-73c6-417d-ba23-e979e0866e2d" containerName="main" Nov 02 20:15:00.190616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.190081 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1ee935-9e4f-4e64-8e87-7821569346cc" containerName="istio-proxy" Nov 02 20:15:00.190616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.190086 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1c21e9-8348-4d6d-bff8-fda75236b9e3" containerName="payload-processing" Nov 02 20:15:00.190616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.190094 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" Nov 02 20:15:00.190616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.190102 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b178ea-5ea9-4de5-a01d-a4a0ceda2eb0" containerName="istio-proxy" Nov 02 20:15:00.190616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.190109 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad146ede-1b0e-4d32-b7bd-c95590295f34" containerName="istio-proxy" Nov 02 20:15:00.190616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.190117 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="533d8bea-d517-4550-b754-85bedfa0200d" containerName="istio-proxy" Nov 02 20:15:00.192988 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.192967 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv"] Nov 02 20:15:00.193149 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.193125 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:00.196768 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.196750 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:00.214742 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.214713 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk"] Nov 02 20:15:00.218584 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.218565 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:00.240748 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.240713 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj"] Nov 02 20:15:00.244704 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.244679 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc"] Nov 02 20:15:00.244851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.244814 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:00.248174 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.248152 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:00.263807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.263784 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:15:00.318100 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.318061 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv"] Nov 02 20:15:00.351249 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351208 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddgg\" (UniqueName: \"kubernetes.io/projected/1445634e-bc86-41d4-972c-a61e056dc60b-kube-api-access-gddgg\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk\" (UID: \"1445634e-bc86-41d4-972c-a61e056dc60b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:00.351445 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351265 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3061fab2-9827-407a-951f-0220331f0a21-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894175-knlfj\" (UID: \"3061fab2-9827-407a-951f-0220331f0a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:00.351445 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351427 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1445634e-bc86-41d4-972c-a61e056dc60b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk\" (UID: \"1445634e-bc86-41d4-972c-a61e056dc60b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:00.351569 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351530 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ad708-6b20-40a7-84b4-3b22ee9612c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc\" (UID: \"844ad708-6b20-40a7-84b4-3b22ee9612c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:00.351631 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351615 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130949ec-3bf8-456b-be8e-3968339e51a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv\" (UID: \"130949ec-3bf8-456b-be8e-3968339e51a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:00.351689 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351655 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007830fe-7b7f-4f0e-aae1-171927294d93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7\" (UID: \"007830fe-7b7f-4f0e-aae1-171927294d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:00.351770 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351686 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscmf\" (UniqueName: \"kubernetes.io/projected/007830fe-7b7f-4f0e-aae1-171927294d93-kube-api-access-rscmf\") pod \"maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7\" (UID: \"007830fe-7b7f-4f0e-aae1-171927294d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:00.351770 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351714 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mfwl\" (UniqueName: \"kubernetes.io/projected/130949ec-3bf8-456b-be8e-3968339e51a0-kube-api-access-9mfwl\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv\" (UID: \"130949ec-3bf8-456b-be8e-3968339e51a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:00.351899 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351770 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzv8t\" (UniqueName: \"kubernetes.io/projected/3061fab2-9827-407a-951f-0220331f0a21-kube-api-access-lzv8t\") pod \"maas-api-key-cleanup-29894175-knlfj\" (UID: \"3061fab2-9827-407a-951f-0220331f0a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:00.351899 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.351881 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vjf\" (UniqueName: \"kubernetes.io/projected/844ad708-6b20-40a7-84b4-3b22ee9612c1-kube-api-access-s6vjf\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc\" (UID: \"844ad708-6b20-40a7-84b4-3b22ee9612c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:00.368837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.368794 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk"] Nov 02 20:15:00.369255 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.369233 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc"] Nov 02 20:15:00.427348 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.427315 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7"] Nov 02 20:15:00.453407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453313 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ad708-6b20-40a7-84b4-3b22ee9612c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc\" (UID: \"844ad708-6b20-40a7-84b4-3b22ee9612c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:00.453407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453379 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130949ec-3bf8-456b-be8e-3968339e51a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv\" (UID: \"130949ec-3bf8-456b-be8e-3968339e51a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:00.453696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453419 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007830fe-7b7f-4f0e-aae1-171927294d93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7\" (UID: \"007830fe-7b7f-4f0e-aae1-171927294d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:00.453696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453448 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscmf\" (UniqueName: \"kubernetes.io/projected/007830fe-7b7f-4f0e-aae1-171927294d93-kube-api-access-rscmf\") pod \"maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7\" (UID: \"007830fe-7b7f-4f0e-aae1-171927294d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:00.453696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453489 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mfwl\" (UniqueName: \"kubernetes.io/projected/130949ec-3bf8-456b-be8e-3968339e51a0-kube-api-access-9mfwl\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv\" (UID: \"130949ec-3bf8-456b-be8e-3968339e51a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:00.453696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453539 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzv8t\" (UniqueName: \"kubernetes.io/projected/3061fab2-9827-407a-951f-0220331f0a21-kube-api-access-lzv8t\") pod \"maas-api-key-cleanup-29894175-knlfj\" (UID: \"3061fab2-9827-407a-951f-0220331f0a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:00.453696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453641 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vjf\" (UniqueName: \"kubernetes.io/projected/844ad708-6b20-40a7-84b4-3b22ee9612c1-kube-api-access-s6vjf\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc\" (UID: \"844ad708-6b20-40a7-84b4-3b22ee9612c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:00.453973 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453711 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddgg\" (UniqueName: \"kubernetes.io/projected/1445634e-bc86-41d4-972c-a61e056dc60b-kube-api-access-gddgg\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk\" (UID: \"1445634e-bc86-41d4-972c-a61e056dc60b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:00.453973 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453746 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3061fab2-9827-407a-951f-0220331f0a21-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894175-knlfj\" (UID: \"3061fab2-9827-407a-951f-0220331f0a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:00.453973 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.453843 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1445634e-bc86-41d4-972c-a61e056dc60b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk\" (UID: \"1445634e-bc86-41d4-972c-a61e056dc60b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:00.454353 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.454327 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007830fe-7b7f-4f0e-aae1-171927294d93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7\" (UID: \"007830fe-7b7f-4f0e-aae1-171927294d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:00.454443 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.454354 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130949ec-3bf8-456b-be8e-3968339e51a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv\" (UID: \"130949ec-3bf8-456b-be8e-3968339e51a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:00.454752 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.454726 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3061fab2-9827-407a-951f-0220331f0a21-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894175-knlfj\" (UID: \"3061fab2-9827-407a-951f-0220331f0a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:00.454843 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.454726 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1445634e-bc86-41d4-972c-a61e056dc60b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk\" (UID: \"1445634e-bc86-41d4-972c-a61e056dc60b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:00.454966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.454946 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ad708-6b20-40a7-84b4-3b22ee9612c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc\" (UID: \"844ad708-6b20-40a7-84b4-3b22ee9612c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:00.489817 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.489777 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj"] Nov 02 20:15:00.513097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.512845 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzv8t\" (UniqueName: \"kubernetes.io/projected/3061fab2-9827-407a-951f-0220331f0a21-kube-api-access-lzv8t\") pod \"maas-api-key-cleanup-29894175-knlfj\" (UID: \"3061fab2-9827-407a-951f-0220331f0a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:00.513097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.512945 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddgg\" (UniqueName: \"kubernetes.io/projected/1445634e-bc86-41d4-972c-a61e056dc60b-kube-api-access-gddgg\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk\" (UID: \"1445634e-bc86-41d4-972c-a61e056dc60b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:00.513514 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.513488 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vjf\" (UniqueName: \"kubernetes.io/projected/844ad708-6b20-40a7-84b4-3b22ee9612c1-kube-api-access-s6vjf\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc\" (UID: \"844ad708-6b20-40a7-84b4-3b22ee9612c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:00.513606 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.513584 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscmf\" (UniqueName: \"kubernetes.io/projected/007830fe-7b7f-4f0e-aae1-171927294d93-kube-api-access-rscmf\") pod \"maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7\" (UID: \"007830fe-7b7f-4f0e-aae1-171927294d93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:00.514073 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.514046 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mfwl\" (UniqueName: \"kubernetes.io/projected/130949ec-3bf8-456b-be8e-3968339e51a0-kube-api-access-9mfwl\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv\" (UID: \"130949ec-3bf8-456b-be8e-3968339e51a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:00.515887 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.515864 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:00.532844 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.532813 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:00.560052 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.560015 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:00.566902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.566869 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:00.808366 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.808287 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:00.904713 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.904681 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv"] Nov 02 20:15:00.906231 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:15:00.906207 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130949ec_3bf8_456b_be8e_3968339e51a0.slice/crio-34d3e270b7657f1defbba8155f3f12805b636ea38100c15e85530e142351b25d WatchSource:0}: Error finding container 34d3e270b7657f1defbba8155f3f12805b636ea38100c15e85530e142351b25d: Status 404 returned error can't find the container with id 34d3e270b7657f1defbba8155f3f12805b636ea38100c15e85530e142351b25d Nov 02 20:15:00.952290 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:15:00.952243 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1445634e_bc86_41d4_972c_a61e056dc60b.slice/crio-f1b4d3c6c21b02dbf447646415f8b697b10b3a3a6e52d9e90247c57db2739f5b WatchSource:0}: Error finding container f1b4d3c6c21b02dbf447646415f8b697b10b3a3a6e52d9e90247c57db2739f5b: Status 404 returned error can't find the container with id f1b4d3c6c21b02dbf447646415f8b697b10b3a3a6e52d9e90247c57db2739f5b Nov 02 20:15:00.954013 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:00.953987 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk"] Nov 02 20:15:01.004832 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:15:01.004798 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3061fab2_9827_407a_951f_0220331f0a21.slice/crio-dc91976458637c2ae1dd8815a846c1f301010eb835173722ee7a37464fa0de82 WatchSource:0}: Error finding container dc91976458637c2ae1dd8815a846c1f301010eb835173722ee7a37464fa0de82: Status 404 returned error can't find the container with id dc91976458637c2ae1dd8815a846c1f301010eb835173722ee7a37464fa0de82 Nov 02 20:15:01.007049 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:15:01.007020 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ad708_6b20_40a7_84b4_3b22ee9612c1.slice/crio-c55ec45d0dfee8a9b2a95105714335bcb77c9ed7e24efc20e46ae32006ca8fbf WatchSource:0}: Error finding container c55ec45d0dfee8a9b2a95105714335bcb77c9ed7e24efc20e46ae32006ca8fbf: Status 404 returned error can't find the container with id c55ec45d0dfee8a9b2a95105714335bcb77c9ed7e24efc20e46ae32006ca8fbf Nov 02 20:15:01.007344 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.007315 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj"] Nov 02 20:15:01.015790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.011708 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc"] Nov 02 20:15:01.183031 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:15:01.182994 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007830fe_7b7f_4f0e_aae1_171927294d93.slice/crio-b29b7101ac60f151d7b68ce3db477911a7e4ba6a5af04e247471adcca813a5f5 WatchSource:0}: Error finding container b29b7101ac60f151d7b68ce3db477911a7e4ba6a5af04e247471adcca813a5f5: Status 404 returned error can't find the container with id b29b7101ac60f151d7b68ce3db477911a7e4ba6a5af04e247471adcca813a5f5 Nov 02 20:15:01.185339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.185313 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7"] Nov 02 20:15:01.537797 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.537756 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" event={"ID":"007830fe-7b7f-4f0e-aae1-171927294d93","Type":"ContainerStarted","Data":"b29b7101ac60f151d7b68ce3db477911a7e4ba6a5af04e247471adcca813a5f5"} Nov 02 20:15:01.539268 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.539237 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" event={"ID":"3061fab2-9827-407a-951f-0220331f0a21","Type":"ContainerStarted","Data":"dc91976458637c2ae1dd8815a846c1f301010eb835173722ee7a37464fa0de82"} Nov 02 20:15:01.541065 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.541039 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" event={"ID":"1445634e-bc86-41d4-972c-a61e056dc60b","Type":"ContainerStarted","Data":"ec56fdfa4cbd182e5642193044e4f1a86f6a1c19be8ffda52e06836e91aae267"} Nov 02 20:15:01.541198 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.541084 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" event={"ID":"1445634e-bc86-41d4-972c-a61e056dc60b","Type":"ContainerStarted","Data":"f1b4d3c6c21b02dbf447646415f8b697b10b3a3a6e52d9e90247c57db2739f5b"} Nov 02 20:15:01.542907 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.542876 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" event={"ID":"844ad708-6b20-40a7-84b4-3b22ee9612c1","Type":"ContainerStarted","Data":"812b6c82bed408dd23688da34ab95744afa202ba0af058deee4470857a1fb9e2"} Nov 02 20:15:01.543042 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.542922 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" event={"ID":"844ad708-6b20-40a7-84b4-3b22ee9612c1","Type":"ContainerStarted","Data":"c55ec45d0dfee8a9b2a95105714335bcb77c9ed7e24efc20e46ae32006ca8fbf"} Nov 02 20:15:01.544590 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.544567 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" event={"ID":"130949ec-3bf8-456b-be8e-3968339e51a0","Type":"ContainerStarted","Data":"da0fc3f84860deff8cb3758c3079c12518b83e208867112767c84b4e90515708"} Nov 02 20:15:01.544718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.544608 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" event={"ID":"130949ec-3bf8-456b-be8e-3968339e51a0","Type":"ContainerStarted","Data":"34d3e270b7657f1defbba8155f3f12805b636ea38100c15e85530e142351b25d"} Nov 02 20:15:01.658352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.658268 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" podStartSLOduration=1.658249338 podStartE2EDuration="1.658249338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:01.657232302 +0000 UTC m=+2759.999621382" watchObservedRunningTime="2026-11-02 20:15:01.658249338 +0000 UTC m=+2760.000638420" Nov 02 20:15:01.789000 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.788895 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" podStartSLOduration=1.7888796660000001 podStartE2EDuration="1.788879666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:01.785943382 +0000 UTC m=+2760.128332464" watchObservedRunningTime="2026-11-02 20:15:01.788879666 +0000 UTC m=+2760.131268724" Nov 02 20:15:01.903941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:01.903884 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" podStartSLOduration=1.903867505 podStartE2EDuration="1.903867505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:01.902066136 +0000 UTC m=+2760.244455216" watchObservedRunningTime="2026-11-02 20:15:01.903867505 +0000 UTC m=+2760.246256626" Nov 02 20:15:02.450669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.450628 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:02.554046 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.554014 5850 generic.go:356] "Generic (PLEG): container finished" podID="007830fe-7b7f-4f0e-aae1-171927294d93" containerID="f4416fc94eeda693dd0feafabe2ff8dfcebb30b6621513aa00b67b62fe9708c8" exitCode=60 Nov 02 20:15:02.554511 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.554110 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" event={"ID":"007830fe-7b7f-4f0e-aae1-171927294d93","Type":"ContainerDied","Data":"f4416fc94eeda693dd0feafabe2ff8dfcebb30b6621513aa00b67b62fe9708c8"} Nov 02 20:15:02.554511 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.554404 5850 scope.go:122] "RemoveContainer" containerID="f4416fc94eeda693dd0feafabe2ff8dfcebb30b6621513aa00b67b62fe9708c8" Nov 02 20:15:02.555940 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.555915 5850 generic.go:356] "Generic (PLEG): container finished" podID="3061fab2-9827-407a-951f-0220331f0a21" containerID="5fe19445ad8002a878dd5afb7c3a2a7d8664ee3a8493f1694190315bffd1fc72" exitCode=60 Nov 02 20:15:02.556068 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.555942 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" event={"ID":"3061fab2-9827-407a-951f-0220331f0a21","Type":"ContainerDied","Data":"5fe19445ad8002a878dd5afb7c3a2a7d8664ee3a8493f1694190315bffd1fc72"} Nov 02 20:15:02.556282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.556264 5850 scope.go:122] "RemoveContainer" containerID="5fe19445ad8002a878dd5afb7c3a2a7d8664ee3a8493f1694190315bffd1fc72" Nov 02 20:15:02.558093 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.557901 5850 generic.go:356] "Generic (PLEG): container finished" podID="1445634e-bc86-41d4-972c-a61e056dc60b" containerID="ec56fdfa4cbd182e5642193044e4f1a86f6a1c19be8ffda52e06836e91aae267" exitCode=60 Nov 02 20:15:02.558093 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.557980 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" event={"ID":"1445634e-bc86-41d4-972c-a61e056dc60b","Type":"ContainerDied","Data":"ec56fdfa4cbd182e5642193044e4f1a86f6a1c19be8ffda52e06836e91aae267"} Nov 02 20:15:02.558255 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.558237 5850 scope.go:122] "RemoveContainer" containerID="ec56fdfa4cbd182e5642193044e4f1a86f6a1c19be8ffda52e06836e91aae267" Nov 02 20:15:02.559754 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.559734 5850 generic.go:356] "Generic (PLEG): container finished" podID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerID="812b6c82bed408dd23688da34ab95744afa202ba0af058deee4470857a1fb9e2" exitCode=60 Nov 02 20:15:02.559848 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.559765 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" event={"ID":"844ad708-6b20-40a7-84b4-3b22ee9612c1","Type":"ContainerDied","Data":"812b6c82bed408dd23688da34ab95744afa202ba0af058deee4470857a1fb9e2"} Nov 02 20:15:02.560059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.560041 5850 scope.go:122] "RemoveContainer" containerID="812b6c82bed408dd23688da34ab95744afa202ba0af058deee4470857a1fb9e2" Nov 02 20:15:02.561325 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.561304 5850 generic.go:356] "Generic (PLEG): container finished" podID="130949ec-3bf8-456b-be8e-3968339e51a0" containerID="da0fc3f84860deff8cb3758c3079c12518b83e208867112767c84b4e90515708" exitCode=60 Nov 02 20:15:02.561399 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.561341 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" event={"ID":"130949ec-3bf8-456b-be8e-3968339e51a0","Type":"ContainerDied","Data":"da0fc3f84860deff8cb3758c3079c12518b83e208867112767c84b4e90515708"} Nov 02 20:15:02.561574 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:02.561561 5850 scope.go:122] "RemoveContainer" containerID="da0fc3f84860deff8cb3758c3079c12518b83e208867112767c84b4e90515708" Nov 02 20:15:03.570121 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:03.570077 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" event={"ID":"844ad708-6b20-40a7-84b4-3b22ee9612c1","Type":"ContainerStarted","Data":"36cb4f9078a841f7e14df70b549560f476bc100717d66fe93dd8914222a6a8a4"} Nov 02 20:15:03.581418 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:03.581376 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" event={"ID":"130949ec-3bf8-456b-be8e-3968339e51a0","Type":"ContainerStarted","Data":"2c47976e3b95604c8b59e22c461fbda266e6e00898b50c09bf93f5e0a3cdb9f8"} Nov 02 20:15:03.583614 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:03.583580 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" event={"ID":"007830fe-7b7f-4f0e-aae1-171927294d93","Type":"ContainerStarted","Data":"2b5c4580f290309dc979e16d00799d37b08a4ca2f4c0b97e41c516a290fb28a5"} Nov 02 20:15:03.585656 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:03.585631 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" event={"ID":"3061fab2-9827-407a-951f-0220331f0a21","Type":"ContainerStarted","Data":"b7b1adfae244c174537183605a3fca537c800e2ab825160d5db047e4104a27df"} Nov 02 20:15:03.587414 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:03.587388 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" event={"ID":"1445634e-bc86-41d4-972c-a61e056dc60b","Type":"ContainerStarted","Data":"a5d690142c902f4bd3c8f6481f13f27d56fb309bf2e47c066f72d2844d169aa7"} Nov 02 20:15:03.920825 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:03.920769 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" podStartSLOduration=3.920751505 podStartE2EDuration="3.920751505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:03.918404045 +0000 UTC m=+2762.260793119" watchObservedRunningTime="2026-11-02 20:15:03.920751505 +0000 UTC m=+2762.263140578" Nov 02 20:15:04.154058 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.153996 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" podStartSLOduration=4.153978099 podStartE2EDuration="4.153978099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:04.152728036 +0000 UTC m=+2762.495117113" watchObservedRunningTime="2026-11-02 20:15:04.153978099 +0000 UTC m=+2762.496367180" Nov 02 20:15:04.596630 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.596544 5850 generic.go:356] "Generic (PLEG): container finished" podID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerID="36cb4f9078a841f7e14df70b549560f476bc100717d66fe93dd8914222a6a8a4" exitCode=60 Nov 02 20:15:04.596630 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.596608 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" event={"ID":"844ad708-6b20-40a7-84b4-3b22ee9612c1","Type":"ContainerDied","Data":"36cb4f9078a841f7e14df70b549560f476bc100717d66fe93dd8914222a6a8a4"} Nov 02 20:15:04.597125 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.596672 5850 scope.go:122] "RemoveContainer" containerID="812b6c82bed408dd23688da34ab95744afa202ba0af058deee4470857a1fb9e2" Nov 02 20:15:04.597125 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.596987 5850 scope.go:122] "RemoveContainer" containerID="36cb4f9078a841f7e14df70b549560f476bc100717d66fe93dd8914222a6a8a4" Nov 02 20:15:04.597387 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:04.597348 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc_odh-ai-gateway-infra(844ad708-6b20-40a7-84b4-3b22ee9612c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" Nov 02 20:15:04.598672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.598649 5850 generic.go:356] "Generic (PLEG): container finished" podID="130949ec-3bf8-456b-be8e-3968339e51a0" containerID="2c47976e3b95604c8b59e22c461fbda266e6e00898b50c09bf93f5e0a3cdb9f8" exitCode=60 Nov 02 20:15:04.598779 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.598734 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" event={"ID":"130949ec-3bf8-456b-be8e-3968339e51a0","Type":"ContainerDied","Data":"2c47976e3b95604c8b59e22c461fbda266e6e00898b50c09bf93f5e0a3cdb9f8"} Nov 02 20:15:04.598978 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.598962 5850 scope.go:122] "RemoveContainer" containerID="2c47976e3b95604c8b59e22c461fbda266e6e00898b50c09bf93f5e0a3cdb9f8" Nov 02 20:15:04.599283 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:04.599261 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv_odh-ai-gateway-infra(130949ec-3bf8-456b-be8e-3968339e51a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" Nov 02 20:15:04.600684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.600663 5850 generic.go:356] "Generic (PLEG): container finished" podID="007830fe-7b7f-4f0e-aae1-171927294d93" containerID="2b5c4580f290309dc979e16d00799d37b08a4ca2f4c0b97e41c516a290fb28a5" exitCode=60 Nov 02 20:15:04.600795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.600740 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" event={"ID":"007830fe-7b7f-4f0e-aae1-171927294d93","Type":"ContainerDied","Data":"2b5c4580f290309dc979e16d00799d37b08a4ca2f4c0b97e41c516a290fb28a5"} Nov 02 20:15:04.601004 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.600987 5850 scope.go:122] "RemoveContainer" containerID="2b5c4580f290309dc979e16d00799d37b08a4ca2f4c0b97e41c516a290fb28a5" Nov 02 20:15:04.601266 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:04.601244 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7_odh-ai-gateway-infra(007830fe-7b7f-4f0e-aae1-171927294d93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" Nov 02 20:15:04.602704 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.602683 5850 generic.go:356] "Generic (PLEG): container finished" podID="3061fab2-9827-407a-951f-0220331f0a21" containerID="b7b1adfae244c174537183605a3fca537c800e2ab825160d5db047e4104a27df" exitCode=60 Nov 02 20:15:04.602802 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.602741 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" event={"ID":"3061fab2-9827-407a-951f-0220331f0a21","Type":"ContainerDied","Data":"b7b1adfae244c174537183605a3fca537c800e2ab825160d5db047e4104a27df"} Nov 02 20:15:04.603027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.603010 5850 scope.go:122] "RemoveContainer" containerID="b7b1adfae244c174537183605a3fca537c800e2ab825160d5db047e4104a27df" Nov 02 20:15:04.603297 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:04.603279 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894175-knlfj_odh-ai-gateway-infra(3061fab2-9827-407a-951f-0220331f0a21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" podUID="3061fab2-9827-407a-951f-0220331f0a21" Nov 02 20:15:04.604716 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.604698 5850 generic.go:356] "Generic (PLEG): container finished" podID="1445634e-bc86-41d4-972c-a61e056dc60b" containerID="a5d690142c902f4bd3c8f6481f13f27d56fb309bf2e47c066f72d2844d169aa7" exitCode=60 Nov 02 20:15:04.604810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.604766 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" event={"ID":"1445634e-bc86-41d4-972c-a61e056dc60b","Type":"ContainerDied","Data":"a5d690142c902f4bd3c8f6481f13f27d56fb309bf2e47c066f72d2844d169aa7"} Nov 02 20:15:04.605018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.605000 5850 scope.go:122] "RemoveContainer" containerID="a5d690142c902f4bd3c8f6481f13f27d56fb309bf2e47c066f72d2844d169aa7" Nov 02 20:15:04.605242 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:04.605225 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk_odh-ai-gateway-infra(1445634e-bc86-41d4-972c-a61e056dc60b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" Nov 02 20:15:04.610043 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.610025 5850 scope.go:122] "RemoveContainer" containerID="da0fc3f84860deff8cb3758c3079c12518b83e208867112767c84b4e90515708" Nov 02 20:15:04.624612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.624586 5850 scope.go:122] "RemoveContainer" containerID="f4416fc94eeda693dd0feafabe2ff8dfcebb30b6621513aa00b67b62fe9708c8" Nov 02 20:15:04.638503 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.638453 5850 scope.go:122] "RemoveContainer" containerID="5fe19445ad8002a878dd5afb7c3a2a7d8664ee3a8493f1694190315bffd1fc72" Nov 02 20:15:04.651950 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:04.651928 5850 scope.go:122] "RemoveContainer" containerID="ec56fdfa4cbd182e5642193044e4f1a86f6a1c19be8ffda52e06836e91aae267" Nov 02 20:15:05.613162 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:05.613131 5850 scope.go:122] "RemoveContainer" containerID="36cb4f9078a841f7e14df70b549560f476bc100717d66fe93dd8914222a6a8a4" Nov 02 20:15:05.613755 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:05.613515 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc_odh-ai-gateway-infra(844ad708-6b20-40a7-84b4-3b22ee9612c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" Nov 02 20:15:05.614991 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:05.614966 5850 scope.go:122] "RemoveContainer" containerID="2c47976e3b95604c8b59e22c461fbda266e6e00898b50c09bf93f5e0a3cdb9f8" Nov 02 20:15:05.615318 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:05.615287 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv_odh-ai-gateway-infra(130949ec-3bf8-456b-be8e-3968339e51a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" Nov 02 20:15:05.616521 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:05.616499 5850 scope.go:122] "RemoveContainer" containerID="2b5c4580f290309dc979e16d00799d37b08a4ca2f4c0b97e41c516a290fb28a5" Nov 02 20:15:05.616793 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:05.616774 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7_odh-ai-gateway-infra(007830fe-7b7f-4f0e-aae1-171927294d93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" Nov 02 20:15:05.618104 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:05.618083 5850 scope.go:122] "RemoveContainer" containerID="b7b1adfae244c174537183605a3fca537c800e2ab825160d5db047e4104a27df" Nov 02 20:15:05.618363 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:05.618342 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894175-knlfj_odh-ai-gateway-infra(3061fab2-9827-407a-951f-0220331f0a21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" podUID="3061fab2-9827-407a-951f-0220331f0a21" Nov 02 20:15:05.619608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:05.619591 5850 scope.go:122] "RemoveContainer" containerID="a5d690142c902f4bd3c8f6481f13f27d56fb309bf2e47c066f72d2844d169aa7" Nov 02 20:15:05.619799 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:05.619781 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk_odh-ai-gateway-infra(1445634e-bc86-41d4-972c-a61e056dc60b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" Nov 02 20:15:06.954762 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:06.954727 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:09.697371 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:09.697335 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:09.858059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:09.858025 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:10.714042 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:10.714007 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr"] Nov 02 20:15:10.739792 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:10.739743 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:10.908223 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:10.908036 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:12.148341 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:12.148281 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:12.406634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:12.406511 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:12.615424 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:12.615387 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:13.076808 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:13.076770 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:13.816388 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:13.816360 5850 scope.go:122] "RemoveContainer" containerID="b7b1adfae244c174537183605a3fca537c800e2ab825160d5db047e4104a27df" Nov 02 20:15:14.706004 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:14.705962 5850 generic.go:356] "Generic (PLEG): container finished" podID="3061fab2-9827-407a-951f-0220331f0a21" containerID="b12e84ba2cd3151253b1ff8faddfc922ee7aeb21eed736b2ec801502399ba57a" exitCode=60 Nov 02 20:15:14.706190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:14.706026 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" event={"ID":"3061fab2-9827-407a-951f-0220331f0a21","Type":"ContainerDied","Data":"b12e84ba2cd3151253b1ff8faddfc922ee7aeb21eed736b2ec801502399ba57a"} Nov 02 20:15:14.706190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:14.706073 5850 scope.go:122] "RemoveContainer" containerID="b7b1adfae244c174537183605a3fca537c800e2ab825160d5db047e4104a27df" Nov 02 20:15:14.706514 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:14.706488 5850 scope.go:122] "RemoveContainer" containerID="b12e84ba2cd3151253b1ff8faddfc922ee7aeb21eed736b2ec801502399ba57a" Nov 02 20:15:14.706872 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:14.706851 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894175-knlfj_odh-ai-gateway-infra(3061fab2-9827-407a-951f-0220331f0a21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" podUID="3061fab2-9827-407a-951f-0220331f0a21" Nov 02 20:15:14.762772 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:14.762735 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:15:15.808324 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:15.808287 5850 scope.go:122] "RemoveContainer" containerID="a5d690142c902f4bd3c8f6481f13f27d56fb309bf2e47c066f72d2844d169aa7" Nov 02 20:15:15.832255 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:15.832226 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj"] Nov 02 20:15:15.981841 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:15.981813 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:16.085283 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.085189 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3061fab2-9827-407a-951f-0220331f0a21-kube-api-access-lzv8t\" (UniqueName: \"kubernetes.io/projected/3061fab2-9827-407a-951f-0220331f0a21-kube-api-access-lzv8t\") pod \"3061fab2-9827-407a-951f-0220331f0a21\" (UID: \"3061fab2-9827-407a-951f-0220331f0a21\") " Nov 02 20:15:16.085283 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.085258 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3061fab2-9827-407a-951f-0220331f0a21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3061fab2-9827-407a-951f-0220331f0a21-maas-api-service-ca\") pod \"3061fab2-9827-407a-951f-0220331f0a21\" (UID: \"3061fab2-9827-407a-951f-0220331f0a21\") " Nov 02 20:15:16.085738 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.085710 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3061fab2-9827-407a-951f-0220331f0a21-maas-api-service-ca" pod "3061fab2-9827-407a-951f-0220331f0a21" (UID: "3061fab2-9827-407a-951f-0220331f0a21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:16.087391 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.087368 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3061fab2-9827-407a-951f-0220331f0a21-kube-api-access-lzv8t" pod "3061fab2-9827-407a-951f-0220331f0a21" (UID: "3061fab2-9827-407a-951f-0220331f0a21"). InnerVolumeSpecName "kube-api-access-lzv8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:16.186867 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.186828 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3061fab2-9827-407a-951f-0220331f0a21-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:16.186867 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.186857 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzv8t\" (UniqueName: \"kubernetes.io/projected/3061fab2-9827-407a-951f-0220331f0a21-kube-api-access-lzv8t\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:16.721599 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.721575 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" Nov 02 20:15:16.721781 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.721575 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj" event={"ID":"3061fab2-9827-407a-951f-0220331f0a21","Type":"ContainerDied","Data":"dc91976458637c2ae1dd8815a846c1f301010eb835173722ee7a37464fa0de82"} Nov 02 20:15:16.721781 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.721719 5850 scope.go:122] "RemoveContainer" containerID="b12e84ba2cd3151253b1ff8faddfc922ee7aeb21eed736b2ec801502399ba57a" Nov 02 20:15:16.724277 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.724251 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" event={"ID":"1445634e-bc86-41d4-972c-a61e056dc60b","Type":"ContainerStarted","Data":"8fb7422efa38602957ecb9541b1f13b94676a69f4f99ca9fa78a178a9e676528"} Nov 02 20:15:16.808086 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.808059 5850 scope.go:122] "RemoveContainer" containerID="2b5c4580f290309dc979e16d00799d37b08a4ca2f4c0b97e41c516a290fb28a5" Nov 02 20:15:16.891692 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.891655 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj"] Nov 02 20:15:16.894030 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:16.893997 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-knlfj"] Nov 02 20:15:17.149780 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.149747 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:17.731879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.731849 5850 generic.go:356] "Generic (PLEG): container finished" podID="007830fe-7b7f-4f0e-aae1-171927294d93" containerID="a1905bd4f10a3abbdc25de40611b688320e6fe08e4df0b7185e335445f7579c1" exitCode=60 Nov 02 20:15:17.732036 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.731923 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" event={"ID":"007830fe-7b7f-4f0e-aae1-171927294d93","Type":"ContainerDied","Data":"a1905bd4f10a3abbdc25de40611b688320e6fe08e4df0b7185e335445f7579c1"} Nov 02 20:15:17.732036 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.731984 5850 scope.go:122] "RemoveContainer" containerID="2b5c4580f290309dc979e16d00799d37b08a4ca2f4c0b97e41c516a290fb28a5" Nov 02 20:15:17.732401 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.732383 5850 scope.go:122] "RemoveContainer" containerID="a1905bd4f10a3abbdc25de40611b688320e6fe08e4df0b7185e335445f7579c1" Nov 02 20:15:17.732766 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:17.732746 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7_odh-ai-gateway-infra(007830fe-7b7f-4f0e-aae1-171927294d93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" Nov 02 20:15:17.734926 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.734906 5850 generic.go:356] "Generic (PLEG): container finished" podID="1445634e-bc86-41d4-972c-a61e056dc60b" containerID="8fb7422efa38602957ecb9541b1f13b94676a69f4f99ca9fa78a178a9e676528" exitCode=60 Nov 02 20:15:17.735031 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.734952 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" event={"ID":"1445634e-bc86-41d4-972c-a61e056dc60b","Type":"ContainerDied","Data":"8fb7422efa38602957ecb9541b1f13b94676a69f4f99ca9fa78a178a9e676528"} Nov 02 20:15:17.735333 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.735315 5850 scope.go:122] "RemoveContainer" containerID="8fb7422efa38602957ecb9541b1f13b94676a69f4f99ca9fa78a178a9e676528" Nov 02 20:15:17.735643 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:17.735619 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk_odh-ai-gateway-infra(1445634e-bc86-41d4-972c-a61e056dc60b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" Nov 02 20:15:17.775242 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.775212 5850 scope.go:122] "RemoveContainer" containerID="a5d690142c902f4bd3c8f6481f13f27d56fb309bf2e47c066f72d2844d169aa7" Nov 02 20:15:17.808652 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.808615 5850 scope.go:122] "RemoveContainer" containerID="2c47976e3b95604c8b59e22c461fbda266e6e00898b50c09bf93f5e0a3cdb9f8" Nov 02 20:15:17.808836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.808673 5850 scope.go:122] "RemoveContainer" containerID="36cb4f9078a841f7e14df70b549560f476bc100717d66fe93dd8914222a6a8a4" Nov 02 20:15:17.818010 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:17.817980 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3061fab2-9827-407a-951f-0220331f0a21" path="/var/lib/kubelet/pods/3061fab2-9827-407a-951f-0220331f0a21/volumes" Nov 02 20:15:18.016191 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:18.016102 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk"] Nov 02 20:15:18.743500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:18.743434 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" event={"ID":"130949ec-3bf8-456b-be8e-3968339e51a0","Type":"ContainerStarted","Data":"106e70820e66f9eecfe6a0c5763e7273018cc5cef6845184253131ffd4b5a24c"} Nov 02 20:15:18.750181 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:18.749515 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" event={"ID":"844ad708-6b20-40a7-84b4-3b22ee9612c1","Type":"ContainerStarted","Data":"11417b8ea7fe7c4024039c32df9d61d7d7d806a7e3cf1ea0e7d05f5acffe5ede"} Nov 02 20:15:18.858594 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:18.858556 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7"] Nov 02 20:15:18.880773 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:18.880726 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:18.901766 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:18.901730 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:19.025546 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.025456 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1445634e-bc86-41d4-972c-a61e056dc60b-kube-api-access-gddgg\" (UniqueName: \"kubernetes.io/projected/1445634e-bc86-41d4-972c-a61e056dc60b-kube-api-access-gddgg\") pod \"1445634e-bc86-41d4-972c-a61e056dc60b\" (UID: \"1445634e-bc86-41d4-972c-a61e056dc60b\") " Nov 02 20:15:19.026078 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.025612 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1445634e-bc86-41d4-972c-a61e056dc60b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1445634e-bc86-41d4-972c-a61e056dc60b-maas-api-service-ca\") pod \"1445634e-bc86-41d4-972c-a61e056dc60b\" (UID: \"1445634e-bc86-41d4-972c-a61e056dc60b\") " Nov 02 20:15:19.026078 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.026024 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1445634e-bc86-41d4-972c-a61e056dc60b-maas-api-service-ca" pod "1445634e-bc86-41d4-972c-a61e056dc60b" (UID: "1445634e-bc86-41d4-972c-a61e056dc60b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:19.027725 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.027702 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1445634e-bc86-41d4-972c-a61e056dc60b-kube-api-access-gddgg" pod "1445634e-bc86-41d4-972c-a61e056dc60b" (UID: "1445634e-bc86-41d4-972c-a61e056dc60b"). InnerVolumeSpecName "kube-api-access-gddgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:19.061179 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.061152 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:19.127780 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.127738 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gddgg\" (UniqueName: \"kubernetes.io/projected/1445634e-bc86-41d4-972c-a61e056dc60b-kube-api-access-gddgg\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.127780 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.127777 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1445634e-bc86-41d4-972c-a61e056dc60b-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.228252 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.228211 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007830fe-7b7f-4f0e-aae1-171927294d93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007830fe-7b7f-4f0e-aae1-171927294d93-maas-api-service-ca\") pod \"007830fe-7b7f-4f0e-aae1-171927294d93\" (UID: \"007830fe-7b7f-4f0e-aae1-171927294d93\") " Nov 02 20:15:19.228457 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.228308 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007830fe-7b7f-4f0e-aae1-171927294d93-kube-api-access-rscmf\" (UniqueName: \"kubernetes.io/projected/007830fe-7b7f-4f0e-aae1-171927294d93-kube-api-access-rscmf\") pod \"007830fe-7b7f-4f0e-aae1-171927294d93\" (UID: \"007830fe-7b7f-4f0e-aae1-171927294d93\") " Nov 02 20:15:19.228605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.228580 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007830fe-7b7f-4f0e-aae1-171927294d93-maas-api-service-ca" pod "007830fe-7b7f-4f0e-aae1-171927294d93" (UID: "007830fe-7b7f-4f0e-aae1-171927294d93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:19.228739 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.228724 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007830fe-7b7f-4f0e-aae1-171927294d93-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.230448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.230426 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007830fe-7b7f-4f0e-aae1-171927294d93-kube-api-access-rscmf" pod "007830fe-7b7f-4f0e-aae1-171927294d93" (UID: "007830fe-7b7f-4f0e-aae1-171927294d93"). InnerVolumeSpecName "kube-api-access-rscmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:19.290877 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.290792 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:19.330827 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.330670 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rscmf\" (UniqueName: \"kubernetes.io/projected/007830fe-7b7f-4f0e-aae1-171927294d93-kube-api-access-rscmf\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.454884 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.454848 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:19.655848 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.655815 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:19.779680 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.779638 5850 generic.go:356] "Generic (PLEG): container finished" podID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerID="11417b8ea7fe7c4024039c32df9d61d7d7d806a7e3cf1ea0e7d05f5acffe5ede" exitCode=60 Nov 02 20:15:19.779896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.779759 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" event={"ID":"844ad708-6b20-40a7-84b4-3b22ee9612c1","Type":"ContainerDied","Data":"11417b8ea7fe7c4024039c32df9d61d7d7d806a7e3cf1ea0e7d05f5acffe5ede"} Nov 02 20:15:19.779896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.779822 5850 scope.go:122] "RemoveContainer" containerID="36cb4f9078a841f7e14df70b549560f476bc100717d66fe93dd8914222a6a8a4" Nov 02 20:15:19.780372 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.780342 5850 scope.go:122] "RemoveContainer" containerID="11417b8ea7fe7c4024039c32df9d61d7d7d806a7e3cf1ea0e7d05f5acffe5ede" Nov 02 20:15:19.780812 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:19.780785 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc_odh-ai-gateway-infra(844ad708-6b20-40a7-84b4-3b22ee9612c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" Nov 02 20:15:19.785107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.785080 5850 generic.go:356] "Generic (PLEG): container finished" podID="130949ec-3bf8-456b-be8e-3968339e51a0" containerID="106e70820e66f9eecfe6a0c5763e7273018cc5cef6845184253131ffd4b5a24c" exitCode=60 Nov 02 20:15:19.785226 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.785153 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" event={"ID":"130949ec-3bf8-456b-be8e-3968339e51a0","Type":"ContainerDied","Data":"106e70820e66f9eecfe6a0c5763e7273018cc5cef6845184253131ffd4b5a24c"} Nov 02 20:15:19.785591 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.785574 5850 scope.go:122] "RemoveContainer" containerID="106e70820e66f9eecfe6a0c5763e7273018cc5cef6845184253131ffd4b5a24c" Nov 02 20:15:19.785947 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:19.785926 5850 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv_odh-ai-gateway-infra(130949ec-3bf8-456b-be8e-3968339e51a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" Nov 02 20:15:19.786775 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.786757 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" Nov 02 20:15:19.786860 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.786781 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7" event={"ID":"007830fe-7b7f-4f0e-aae1-171927294d93","Type":"ContainerDied","Data":"b29b7101ac60f151d7b68ce3db477911a7e4ba6a5af04e247471adcca813a5f5"} Nov 02 20:15:19.793228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.793191 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" event={"ID":"1445634e-bc86-41d4-972c-a61e056dc60b","Type":"ContainerDied","Data":"f1b4d3c6c21b02dbf447646415f8b697b10b3a3a6e52d9e90247c57db2739f5b"} Nov 02 20:15:19.793375 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.793280 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk" Nov 02 20:15:19.797113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.797033 5850 scope.go:122] "RemoveContainer" containerID="2c47976e3b95604c8b59e22c461fbda266e6e00898b50c09bf93f5e0a3cdb9f8" Nov 02 20:15:19.811057 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.811034 5850 scope.go:122] "RemoveContainer" containerID="a1905bd4f10a3abbdc25de40611b688320e6fe08e4df0b7185e335445f7579c1" Nov 02 20:15:19.820293 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.820271 5850 scope.go:122] "RemoveContainer" containerID="8fb7422efa38602957ecb9541b1f13b94676a69f4f99ca9fa78a178a9e676528" Nov 02 20:15:19.869841 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:19.869802 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv"] Nov 02 20:15:20.101995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:20.101905 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc"] Nov 02 20:15:20.103931 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:20.103894 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:20.130710 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:20.130671 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7"] Nov 02 20:15:20.138723 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:20.138684 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5b64d-29894175-djdg7"] Nov 02 20:15:20.250550 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:20.250514 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk"] Nov 02 20:15:20.252720 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:20.252697 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894175-lckvk"] Nov 02 20:15:20.974861 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:20.974834 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:20.976979 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:20.976951 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:21.156386 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.156351 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844ad708-6b20-40a7-84b4-3b22ee9612c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ad708-6b20-40a7-84b4-3b22ee9612c1-maas-api-service-ca\") pod \"844ad708-6b20-40a7-84b4-3b22ee9612c1\" (UID: \"844ad708-6b20-40a7-84b4-3b22ee9612c1\") " Nov 02 20:15:21.156386 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.156394 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844ad708-6b20-40a7-84b4-3b22ee9612c1-kube-api-access-s6vjf\" (UniqueName: \"kubernetes.io/projected/844ad708-6b20-40a7-84b4-3b22ee9612c1-kube-api-access-s6vjf\") pod \"844ad708-6b20-40a7-84b4-3b22ee9612c1\" (UID: \"844ad708-6b20-40a7-84b4-3b22ee9612c1\") " Nov 02 20:15:21.156929 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.156557 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130949ec-3bf8-456b-be8e-3968339e51a0-kube-api-access-9mfwl\" (UniqueName: \"kubernetes.io/projected/130949ec-3bf8-456b-be8e-3968339e51a0-kube-api-access-9mfwl\") pod \"130949ec-3bf8-456b-be8e-3968339e51a0\" (UID: \"130949ec-3bf8-456b-be8e-3968339e51a0\") " Nov 02 20:15:21.156929 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.156704 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/130949ec-3bf8-456b-be8e-3968339e51a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130949ec-3bf8-456b-be8e-3968339e51a0-maas-api-service-ca\") pod \"130949ec-3bf8-456b-be8e-3968339e51a0\" (UID: \"130949ec-3bf8-456b-be8e-3968339e51a0\") " Nov 02 20:15:21.156929 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.156889 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844ad708-6b20-40a7-84b4-3b22ee9612c1-maas-api-service-ca" pod "844ad708-6b20-40a7-84b4-3b22ee9612c1" (UID: "844ad708-6b20-40a7-84b4-3b22ee9612c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:21.157215 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.157187 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130949ec-3bf8-456b-be8e-3968339e51a0-maas-api-service-ca" pod "130949ec-3bf8-456b-be8e-3968339e51a0" (UID: "130949ec-3bf8-456b-be8e-3968339e51a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:21.157401 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.157379 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130949ec-3bf8-456b-be8e-3968339e51a0-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.157486 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.157404 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ad708-6b20-40a7-84b4-3b22ee9612c1-maas-api-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.161500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.161439 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130949ec-3bf8-456b-be8e-3968339e51a0-kube-api-access-9mfwl" pod "130949ec-3bf8-456b-be8e-3968339e51a0" (UID: "130949ec-3bf8-456b-be8e-3968339e51a0"). InnerVolumeSpecName "kube-api-access-9mfwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:21.161776 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.161736 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844ad708-6b20-40a7-84b4-3b22ee9612c1-kube-api-access-s6vjf" pod "844ad708-6b20-40a7-84b4-3b22ee9612c1" (UID: "844ad708-6b20-40a7-84b4-3b22ee9612c1"). InnerVolumeSpecName "kube-api-access-s6vjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:21.258691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.258654 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6vjf\" (UniqueName: \"kubernetes.io/projected/844ad708-6b20-40a7-84b4-3b22ee9612c1-kube-api-access-s6vjf\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.258691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.258688 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mfwl\" (UniqueName: \"kubernetes.io/projected/130949ec-3bf8-456b-be8e-3968339e51a0-kube-api-access-9mfwl\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.766779 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.766745 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:21.825428 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.825400 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" Nov 02 20:15:21.826254 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.826229 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" path="/var/lib/kubelet/pods/007830fe-7b7f-4f0e-aae1-171927294d93/volumes" Nov 02 20:15:21.826783 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.826763 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" path="/var/lib/kubelet/pods/1445634e-bc86-41d4-972c-a61e056dc60b/volumes" Nov 02 20:15:21.827005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.826990 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" Nov 02 20:15:21.827584 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.827565 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc" event={"ID":"844ad708-6b20-40a7-84b4-3b22ee9612c1","Type":"ContainerDied","Data":"c55ec45d0dfee8a9b2a95105714335bcb77c9ed7e24efc20e46ae32006ca8fbf"} Nov 02 20:15:21.827680 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.827608 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv" event={"ID":"130949ec-3bf8-456b-be8e-3968339e51a0","Type":"ContainerDied","Data":"34d3e270b7657f1defbba8155f3f12805b636ea38100c15e85530e142351b25d"} Nov 02 20:15:21.827680 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.827639 5850 scope.go:122] "RemoveContainer" containerID="11417b8ea7fe7c4024039c32df9d61d7d7d806a7e3cf1ea0e7d05f5acffe5ede" Nov 02 20:15:21.841181 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.841162 5850 scope.go:122] "RemoveContainer" containerID="106e70820e66f9eecfe6a0c5763e7273018cc5cef6845184253131ffd4b5a24c" Nov 02 20:15:21.995145 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.995104 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv"] Nov 02 20:15:21.997319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:21.997290 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894175-qgntv"] Nov 02 20:15:22.108870 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:22.108787 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc"] Nov 02 20:15:22.113510 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:22.113466 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894175-l2zgc"] Nov 02 20:15:23.815793 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:23.815747 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" path="/var/lib/kubelet/pods/130949ec-3bf8-456b-be8e-3968339e51a0/volumes" Nov 02 20:15:23.816612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:23.816589 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" path="/var/lib/kubelet/pods/844ad708-6b20-40a7-84b4-3b22ee9612c1/volumes" Nov 02 20:15:25.023062 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:25.023021 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6"] Nov 02 20:15:25.141430 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:25.141392 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:25.740599 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:25.740557 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" podUID="a1aeb7c5-beb6-4ef8-8701-14f2175a553b" containerName="main" containerID="cri-o://95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe" gracePeriod=45 Nov 02 20:15:26.008511 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.008398 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:15:26.117570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.117537 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-model-cache\") pod \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " Nov 02 20:15:26.118038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.117604 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-home\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-home\") pod \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " Nov 02 20:15:26.118038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.117643 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tmp-dir\") pod \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " Nov 02 20:15:26.118038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.117683 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tls-certs\") pod \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " Nov 02 20:15:26.118038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.117717 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-dshm\") pod \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " Nov 02 20:15:26.118038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.117844 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-kube-api-access-zbrb2\" (UniqueName: \"kubernetes.io/projected/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-kube-api-access-zbrb2\") pod \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\" (UID: \"a1aeb7c5-beb6-4ef8-8701-14f2175a553b\") " Nov 02 20:15:26.118038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.117914 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-home" pod "a1aeb7c5-beb6-4ef8-8701-14f2175a553b" (UID: "a1aeb7c5-beb6-4ef8-8701-14f2175a553b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:26.118038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.117929 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-model-cache" pod "a1aeb7c5-beb6-4ef8-8701-14f2175a553b" (UID: "a1aeb7c5-beb6-4ef8-8701-14f2175a553b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:26.118038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.118000 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tmp-dir" pod "a1aeb7c5-beb6-4ef8-8701-14f2175a553b" (UID: "a1aeb7c5-beb6-4ef8-8701-14f2175a553b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:26.118447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.118363 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:26.118447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.118386 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:26.118447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.118401 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:26.120248 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.120220 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-dshm" pod "a1aeb7c5-beb6-4ef8-8701-14f2175a553b" (UID: "a1aeb7c5-beb6-4ef8-8701-14f2175a553b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:26.120394 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.120235 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tls-certs" pod "a1aeb7c5-beb6-4ef8-8701-14f2175a553b" (UID: "a1aeb7c5-beb6-4ef8-8701-14f2175a553b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:26.120518 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.120384 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-kube-api-access-zbrb2" pod "a1aeb7c5-beb6-4ef8-8701-14f2175a553b" (UID: "a1aeb7c5-beb6-4ef8-8701-14f2175a553b"). InnerVolumeSpecName "kube-api-access-zbrb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:26.219552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.219514 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:26.219552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.219546 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:26.219552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.219558 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbrb2\" (UniqueName: \"kubernetes.io/projected/a1aeb7c5-beb6-4ef8-8701-14f2175a553b-kube-api-access-zbrb2\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:26.411140 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.411105 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:26.860722 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.860629 5850 generic.go:356] "Generic (PLEG): container finished" podID="a1aeb7c5-beb6-4ef8-8701-14f2175a553b" containerID="95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe" exitCode=0 Nov 02 20:15:26.860722 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.860676 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" Nov 02 20:15:26.860722 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.860708 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" event={"ID":"a1aeb7c5-beb6-4ef8-8701-14f2175a553b","Type":"ContainerDied","Data":"95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe"} Nov 02 20:15:26.860955 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.860752 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr" event={"ID":"a1aeb7c5-beb6-4ef8-8701-14f2175a553b","Type":"ContainerDied","Data":"cb7f6ce86f41eec2743f899813cce64ce78958c1d07dc8248fa3e0bd77ac10f9"} Nov 02 20:15:26.860955 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.860775 5850 scope.go:122] "RemoveContainer" containerID="95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe" Nov 02 20:15:26.871582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.871556 5850 scope.go:122] "RemoveContainer" containerID="95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe" Nov 02 20:15:26.871894 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:26.871877 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe\": container with ID starting with 95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe not found: ID does not exist" containerID="95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe" Nov 02 20:15:26.871965 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:26.871903 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe"} err="failed to get container status \"95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe\": rpc error: code = NotFound desc = could not find container \"95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe\": container with ID starting with 95e1c58ac6093ce7bb6a8a4549d4072a249e6d5e3d82eaf26b3d0b1a269899fe not found: ID does not exist" Nov 02 20:15:27.029670 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:27.029631 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr"] Nov 02 20:15:27.080577 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:27.080540 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a5b64d/test-model-a5b64d-kserve-59df48f97f-gg5kr"] Nov 02 20:15:27.815282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:27.815250 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1aeb7c5-beb6-4ef8-8701-14f2175a553b" path="/var/lib/kubelet/pods/a1aeb7c5-beb6-4ef8-8701-14f2175a553b/volumes" Nov 02 20:15:30.291215 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.291176 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm"] Nov 02 20:15:30.291789 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.291368 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" podUID="265c981a-ccad-4e88-aeec-16b5efbce6ad" containerName="maas-api" containerID="cri-o://d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e" gracePeriod=30 Nov 02 20:15:30.424328 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.424295 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:30.582186 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.582160 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:15:30.668545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.667657 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265c981a-ccad-4e88-aeec-16b5efbce6ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265c981a-ccad-4e88-aeec-16b5efbce6ad-openshift-service-ca\") pod \"265c981a-ccad-4e88-aeec-16b5efbce6ad\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " Nov 02 20:15:30.668545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.667726 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls\") pod \"265c981a-ccad-4e88-aeec-16b5efbce6ad\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " Nov 02 20:15:30.668545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.667855 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265c981a-ccad-4e88-aeec-16b5efbce6ad-kube-api-access-tfbwp\" (UniqueName: \"kubernetes.io/projected/265c981a-ccad-4e88-aeec-16b5efbce6ad-kube-api-access-tfbwp\") pod \"265c981a-ccad-4e88-aeec-16b5efbce6ad\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " Nov 02 20:15:30.668545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.667886 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-metrics-certs\") pod \"265c981a-ccad-4e88-aeec-16b5efbce6ad\" (UID: \"265c981a-ccad-4e88-aeec-16b5efbce6ad\") " Nov 02 20:15:30.671904 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.671862 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265c981a-ccad-4e88-aeec-16b5efbce6ad-openshift-service-ca" pod "265c981a-ccad-4e88-aeec-16b5efbce6ad" (UID: "265c981a-ccad-4e88-aeec-16b5efbce6ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:30.675754 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.675699 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-metrics-certs" pod "265c981a-ccad-4e88-aeec-16b5efbce6ad" (UID: "265c981a-ccad-4e88-aeec-16b5efbce6ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:30.676422 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.676395 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls" pod "265c981a-ccad-4e88-aeec-16b5efbce6ad" (UID: "265c981a-ccad-4e88-aeec-16b5efbce6ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:30.678194 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.678149 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265c981a-ccad-4e88-aeec-16b5efbce6ad-kube-api-access-tfbwp" pod "265c981a-ccad-4e88-aeec-16b5efbce6ad" (UID: "265c981a-ccad-4e88-aeec-16b5efbce6ad"). InnerVolumeSpecName "kube-api-access-tfbwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:30.769741 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.769699 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfbwp\" (UniqueName: \"kubernetes.io/projected/265c981a-ccad-4e88-aeec-16b5efbce6ad-kube-api-access-tfbwp\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:30.769741 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.769735 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:30.769741 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.769748 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265c981a-ccad-4e88-aeec-16b5efbce6ad-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:30.770010 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.769760 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265c981a-ccad-4e88-aeec-16b5efbce6ad-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:30.885512 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.885462 5850 generic.go:356] "Generic (PLEG): container finished" podID="265c981a-ccad-4e88-aeec-16b5efbce6ad" containerID="d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e" exitCode=0 Nov 02 20:15:30.885721 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.885553 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" Nov 02 20:15:30.885721 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.885590 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" event={"ID":"265c981a-ccad-4e88-aeec-16b5efbce6ad","Type":"ContainerDied","Data":"d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e"} Nov 02 20:15:30.885721 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.885632 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm" event={"ID":"265c981a-ccad-4e88-aeec-16b5efbce6ad","Type":"ContainerDied","Data":"c8353ff4b147346073f28308d3d101f439c71adb3ee1f7a5fb99f2faa5fb4f4a"} Nov 02 20:15:30.885721 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.885655 5850 scope.go:122] "RemoveContainer" containerID="d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e" Nov 02 20:15:30.895998 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.895977 5850 scope.go:122] "RemoveContainer" containerID="d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e" Nov 02 20:15:30.896332 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:30.896312 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e\": container with ID starting with d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e not found: ID does not exist" containerID="d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e" Nov 02 20:15:30.896390 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:30.896340 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e"} err="failed to get container status \"d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e\": rpc error: code = NotFound desc = could not find container \"d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e\": container with ID starting with d29781b485b8d262fa33485b9b5658ab7661fd3139febdc4e46c95b683f27b1e not found: ID does not exist" Nov 02 20:15:31.053580 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:31.053542 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm"] Nov 02 20:15:31.060561 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:31.060530 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-838bc1-78ffc4947-ghmhm"] Nov 02 20:15:31.469822 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:31.469785 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:31.802081 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:31.801987 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:31.816758 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:31.816722 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265c981a-ccad-4e88-aeec-16b5efbce6ad" path="/var/lib/kubelet/pods/265c981a-ccad-4e88-aeec-16b5efbce6ad/volumes" Nov 02 20:15:32.050374 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:32.050340 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:32.074925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:32.074840 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx"] Nov 02 20:15:32.475895 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:32.475863 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:32.683505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:32.672347 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:32.834167 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:32.834077 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:32.996868 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:32.996829 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:33.287790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:33.287757 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:33.448227 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:33.448194 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:33.973875 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:33.973792 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:37.075555 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.075454 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" podUID="f8e19696-fe64-401f-874e-68e07b37b454" containerName="payload-processing" containerID="cri-o://f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9" gracePeriod=25 Nov 02 20:15:37.428773 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.428742 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 20:15:37.503593 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.503552 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e19696-fe64-401f-874e-68e07b37b454-kube-api-access-r8pdm\" (UniqueName: \"kubernetes.io/projected/f8e19696-fe64-401f-874e-68e07b37b454-kube-api-access-r8pdm\") pod \"f8e19696-fe64-401f-874e-68e07b37b454\" (UID: \"f8e19696-fe64-401f-874e-68e07b37b454\") " Nov 02 20:15:37.503808 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.503659 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e19696-fe64-401f-874e-68e07b37b454-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e19696-fe64-401f-874e-68e07b37b454-plugins-config-volume\") pod \"f8e19696-fe64-401f-874e-68e07b37b454\" (UID: \"f8e19696-fe64-401f-874e-68e07b37b454\") " Nov 02 20:15:37.504062 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.504035 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e19696-fe64-401f-874e-68e07b37b454-plugins-config-volume" pod "f8e19696-fe64-401f-874e-68e07b37b454" (UID: "f8e19696-fe64-401f-874e-68e07b37b454"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:37.505754 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.505729 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e19696-fe64-401f-874e-68e07b37b454-kube-api-access-r8pdm" pod "f8e19696-fe64-401f-874e-68e07b37b454" (UID: "f8e19696-fe64-401f-874e-68e07b37b454"). InnerVolumeSpecName "kube-api-access-r8pdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:37.605690 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.605651 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8e19696-fe64-401f-874e-68e07b37b454-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:37.605690 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.605692 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8pdm\" (UniqueName: \"kubernetes.io/projected/f8e19696-fe64-401f-874e-68e07b37b454-kube-api-access-r8pdm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:37.925096 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.925065 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:37.929087 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.929057 5850 generic.go:356] "Generic (PLEG): container finished" podID="f8e19696-fe64-401f-874e-68e07b37b454" containerID="f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9" exitCode=0 Nov 02 20:15:37.929252 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.929111 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" Nov 02 20:15:37.929252 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.929134 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" event={"ID":"f8e19696-fe64-401f-874e-68e07b37b454","Type":"ContainerDied","Data":"f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9"} Nov 02 20:15:37.929252 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.929183 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx" event={"ID":"f8e19696-fe64-401f-874e-68e07b37b454","Type":"ContainerDied","Data":"11ca2589410e9c35ceaf32cd98521707faef7ab313f08f359af818c43db577eb"} Nov 02 20:15:37.929252 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.929205 5850 scope.go:122] "RemoveContainer" containerID="f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9" Nov 02 20:15:37.938686 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.938663 5850 scope.go:122] "RemoveContainer" containerID="f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9" Nov 02 20:15:37.938990 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:37.938967 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9\": container with ID starting with f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9 not found: ID does not exist" containerID="f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9" Nov 02 20:15:37.939052 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:37.938999 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9"} err="failed to get container status \"f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9\": rpc error: code = NotFound desc = could not find container \"f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9\": container with ID starting with f128ea91a266257e0d0fe86f6bbb0f6d277ab4dfb80f9cfda5199d1270358bd9 not found: ID does not exist" Nov 02 20:15:38.097309 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:38.097216 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx"] Nov 02 20:15:38.100747 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:38.100718 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5c46af-79c6575599-4h7nx"] Nov 02 20:15:38.525537 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:38.525507 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:38.689391 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:38.689361 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:38.869849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:38.869818 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:39.816263 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:39.816223 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e19696-fe64-401f-874e-68e07b37b454" path="/var/lib/kubelet/pods/f8e19696-fe64-401f-874e-68e07b37b454/volumes" Nov 02 20:15:39.980836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:39.980803 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:40.044356 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.044308 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" containerID="cri-o://11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4" gracePeriod=45 Nov 02 20:15:40.117392 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.117358 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": dial tcp 10.134.0.66:8000: connect: connection refused" Nov 02 20:15:40.145244 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.145204 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:40.339114 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.339085 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 20:15:40.348164 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.348130 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc"] Nov 02 20:15:40.348364 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.348344 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" containerID="cri-o://fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db" gracePeriod=30 Nov 02 20:15:40.378310 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.378214 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:40.462505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.457026 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tls-certs\") pod \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " Nov 02 20:15:40.462505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.457135 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-dshm\") pod \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " Nov 02 20:15:40.462505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.457246 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-model-cache\") pod \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " Nov 02 20:15:40.462505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.457336 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef32e82-ae3e-4e7d-b263-68e656cf1784-kube-api-access-kpvm5\" (UniqueName: \"kubernetes.io/projected/8ef32e82-ae3e-4e7d-b263-68e656cf1784-kube-api-access-kpvm5\") pod \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " Nov 02 20:15:40.462505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.457383 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-home\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-home\") pod \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " Nov 02 20:15:40.462505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.457410 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tmp-dir\") pod \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\" (UID: \"8ef32e82-ae3e-4e7d-b263-68e656cf1784\") " Nov 02 20:15:40.465798 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.465755 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tmp-dir" pod "8ef32e82-ae3e-4e7d-b263-68e656cf1784" (UID: "8ef32e82-ae3e-4e7d-b263-68e656cf1784"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.466051 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.466021 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-model-cache" pod "8ef32e82-ae3e-4e7d-b263-68e656cf1784" (UID: "8ef32e82-ae3e-4e7d-b263-68e656cf1784"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.485436 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.485360 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-home" pod "8ef32e82-ae3e-4e7d-b263-68e656cf1784" (UID: "8ef32e82-ae3e-4e7d-b263-68e656cf1784"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.489381 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.488189 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef32e82-ae3e-4e7d-b263-68e656cf1784-kube-api-access-kpvm5" pod "8ef32e82-ae3e-4e7d-b263-68e656cf1784" (UID: "8ef32e82-ae3e-4e7d-b263-68e656cf1784"). InnerVolumeSpecName "kube-api-access-kpvm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:40.492620 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.492582 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-dshm" pod "8ef32e82-ae3e-4e7d-b263-68e656cf1784" (UID: "8ef32e82-ae3e-4e7d-b263-68e656cf1784"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.492796 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.492775 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tls-certs" pod "8ef32e82-ae3e-4e7d-b263-68e656cf1784" (UID: "8ef32e82-ae3e-4e7d-b263-68e656cf1784"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:40.558729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.558696 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.558726 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvm5\" (UniqueName: \"kubernetes.io/projected/8ef32e82-ae3e-4e7d-b263-68e656cf1784-kube-api-access-kpvm5\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.558740 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.558749 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.558757 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef32e82-ae3e-4e7d-b263-68e656cf1784-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.558765 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef32e82-ae3e-4e7d-b263-68e656cf1784-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.997870 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.997829 5850 generic.go:356] "Generic (PLEG): container finished" podID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerID="11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4" exitCode=0 Nov 02 20:15:40.998311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.997881 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" Nov 02 20:15:40.998311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.997913 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" event={"ID":"8ef32e82-ae3e-4e7d-b263-68e656cf1784","Type":"ContainerDied","Data":"11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4"} Nov 02 20:15:40.998311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.997961 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6" event={"ID":"8ef32e82-ae3e-4e7d-b263-68e656cf1784","Type":"ContainerDied","Data":"0442c8336c63a18e4417a2f8b1030d9f29ab66ff6632a45d70d9b1d5294a20b0"} Nov 02 20:15:40.998311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:40.997982 5850 scope.go:122] "RemoveContainer" containerID="11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4" Nov 02 20:15:41.007619 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:41.007593 5850 scope.go:122] "RemoveContainer" containerID="11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4" Nov 02 20:15:41.007890 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:41.007873 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4\": container with ID starting with 11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4 not found: ID does not exist" containerID="11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4" Nov 02 20:15:41.007946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:41.007899 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4"} err="failed to get container status \"11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4\": rpc error: code = NotFound desc = could not find container \"11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4\": container with ID starting with 11ea5a6d8eea5b99f67b89299da195709a3c2d836a6c9711814a5cc6615a9fb4 not found: ID does not exist" Nov 02 20:15:41.166733 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:41.166685 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6"] Nov 02 20:15:41.171575 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:41.171540 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/distinct-2-w3-5c46af-kserve-54cbc6d5b4-4t8x6"] Nov 02 20:15:41.816321 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:41.816286 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" path="/var/lib/kubelet/pods/8ef32e82-ae3e-4e7d-b263-68e656cf1784/volumes" Nov 02 20:15:42.590614 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:42.590577 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:44.762356 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:44.762318 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:15:45.621002 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.620975 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:15:45.721049 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721007 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-podinfo\") pod \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " Nov 02 20:15:45.721049 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721055 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-credential-socket\") pod \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " Nov 02 20:15:45.721317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721080 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-socket\") pod \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " Nov 02 20:15:45.721317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721110 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-certs\") pod \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " Nov 02 20:15:45.721317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721138 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istiod-ca-cert\") pod \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " Nov 02 20:15:45.721317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721221 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-kube-api-access-cwcj6\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-kube-api-access-cwcj6\") pod \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " Nov 02 20:15:45.721317 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721302 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-data\") pod \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " Nov 02 20:15:45.721632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721342 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-envoy\") pod \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " Nov 02 20:15:45.721632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721379 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-credential-socket" pod "a632b5f5-8af8-4ec1-9d4d-7970df12d254" (UID: "a632b5f5-8af8-4ec1-9d4d-7970df12d254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:45.721632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721404 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-token\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-token\") pod \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\" (UID: \"a632b5f5-8af8-4ec1-9d4d-7970df12d254\") " Nov 02 20:15:45.721632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721542 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-socket" pod "a632b5f5-8af8-4ec1-9d4d-7970df12d254" (UID: "a632b5f5-8af8-4ec1-9d4d-7970df12d254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:45.721632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721596 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-certs" pod "a632b5f5-8af8-4ec1-9d4d-7970df12d254" (UID: "a632b5f5-8af8-4ec1-9d4d-7970df12d254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:45.721896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721783 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-data" pod "a632b5f5-8af8-4ec1-9d4d-7970df12d254" (UID: "a632b5f5-8af8-4ec1-9d4d-7970df12d254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:45.721896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721811 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istiod-ca-cert" pod "a632b5f5-8af8-4ec1-9d4d-7970df12d254" (UID: "a632b5f5-8af8-4ec1-9d4d-7970df12d254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:45.722008 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.721992 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.722060 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.722010 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.722060 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.722026 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.722060 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.722041 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.722060 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.722053 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.723611 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.723583 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-podinfo" pod "a632b5f5-8af8-4ec1-9d4d-7970df12d254" (UID: "a632b5f5-8af8-4ec1-9d4d-7970df12d254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:45.723751 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.723697 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-token" pod "a632b5f5-8af8-4ec1-9d4d-7970df12d254" (UID: "a632b5f5-8af8-4ec1-9d4d-7970df12d254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:45.723924 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.723902 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-kube-api-access-cwcj6" pod "a632b5f5-8af8-4ec1-9d4d-7970df12d254" (UID: "a632b5f5-8af8-4ec1-9d4d-7970df12d254"). InnerVolumeSpecName "kube-api-access-cwcj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:45.723973 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.723941 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-envoy" pod "a632b5f5-8af8-4ec1-9d4d-7970df12d254" (UID: "a632b5f5-8af8-4ec1-9d4d-7970df12d254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:45.823148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.823117 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.823148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.823144 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.823148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.823154 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a632b5f5-8af8-4ec1-9d4d-7970df12d254-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.823639 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:45.823163 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwcj6\" (UniqueName: \"kubernetes.io/projected/a632b5f5-8af8-4ec1-9d4d-7970df12d254-kube-api-access-cwcj6\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:46.032401 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.032314 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" Nov 02 20:15:46.032401 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.032340 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" event={"ID":"a632b5f5-8af8-4ec1-9d4d-7970df12d254","Type":"ContainerDied","Data":"fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db"} Nov 02 20:15:46.032401 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.032399 5850 scope.go:122] "RemoveContainer" containerID="fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db" Nov 02 20:15:46.032718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.032303 5850 generic.go:356] "Generic (PLEG): container finished" podID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerID="fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db" exitCode=0 Nov 02 20:15:46.032718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.032524 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc" event={"ID":"a632b5f5-8af8-4ec1-9d4d-7970df12d254","Type":"ContainerDied","Data":"3f6dec780a81fdc3eaaeea8e4abf8556a889a7f83c870c1ca23fbcf493b2a5bb"} Nov 02 20:15:46.043214 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.043192 5850 scope.go:122] "RemoveContainer" containerID="e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6" Nov 02 20:15:46.053099 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.053072 5850 scope.go:122] "RemoveContainer" containerID="fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db" Nov 02 20:15:46.053442 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:46.053414 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db\": container with ID starting with fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db not found: ID does not exist" containerID="fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db" Nov 02 20:15:46.053548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.053454 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db"} err="failed to get container status \"fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db\": rpc error: code = NotFound desc = could not find container \"fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db\": container with ID starting with fb14cb3d619466f7a77f1c5bba2c86fb05c592c03f6a4918aa29b7f1fe1828db not found: ID does not exist" Nov 02 20:15:46.053548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.053508 5850 scope.go:122] "RemoveContainer" containerID="e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6" Nov 02 20:15:46.053777 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:15:46.053762 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6\": container with ID starting with e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6 not found: ID does not exist" containerID="e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6" Nov 02 20:15:46.053827 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.053781 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6"} err="failed to get container status \"e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6\": rpc error: code = NotFound desc = could not find container \"e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6\": container with ID starting with e7b828fe459ec61c1601bb0f9728e561b2a84da771e38a6c4c07649831d7a3e6 not found: ID does not exist" Nov 02 20:15:46.201018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.200974 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc"] Nov 02 20:15:46.203794 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:46.203767 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7025fa-openshift-default-56f55b94df-8gblc"] Nov 02 20:15:47.053343 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:47.053308 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:47.815163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:47.815129 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" path="/var/lib/kubelet/pods/a632b5f5-8af8-4ec1-9d4d-7970df12d254/volumes" Nov 02 20:15:51.582965 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:51.582930 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:51.757279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:51.757223 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:51.918176 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:51.918140 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:53.438985 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:53.438913 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:56.364558 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:56.364520 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c"] Nov 02 20:15:56.539582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:56.539515 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:56.683811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:56.683776 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:56.841556 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:56.841263 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:59.252671 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:59.252635 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:59.435925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:15:59.435889 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:01.188678 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:01.187511 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:01.365408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:01.365364 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" podUID="9bbff140-4b0c-492c-a2ba-7163dd68b5c3" containerName="payload-pre-processing" containerID="cri-o://c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9" gracePeriod=25 Nov 02 20:16:01.559090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:01.558985 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:01.709111 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:01.709070 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:02.651967 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:02.651934 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:03.704043 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:03.704000 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" podUID="9bbff140-4b0c-492c-a2ba-7163dd68b5c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:16:05.545468 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:05.545422 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:08.431278 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:08.431242 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:09.941780 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:09.941743 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:13.703720 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:13.703683 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" podUID="9bbff140-4b0c-492c-a2ba-7163dd68b5c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:16:14.356901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:14.356776 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:14.761138 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:14.761102 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:16:18.688385 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:18.688349 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:19.208785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:19.208752 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:19.605139 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:19.605048 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:21.403194 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:21.403154 5850 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbff140_4b0c_492c_a2ba_7163dd68b5c3.slice/crio-conmon-c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:16:21.635966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:21.635897 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:16:21.700911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:21.700870 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-plugins-config-volume\") pod \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\" (UID: \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\") " Nov 02 20:16:21.701132 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:21.701034 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-kube-api-access-gx24g\" (UniqueName: \"kubernetes.io/projected/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-kube-api-access-gx24g\") pod \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\" (UID: \"9bbff140-4b0c-492c-a2ba-7163dd68b5c3\") " Nov 02 20:16:21.701340 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:21.701315 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-plugins-config-volume" pod "9bbff140-4b0c-492c-a2ba-7163dd68b5c3" (UID: "9bbff140-4b0c-492c-a2ba-7163dd68b5c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:21.701511 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:21.701464 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:21.703151 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:21.703129 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-kube-api-access-gx24g" pod "9bbff140-4b0c-492c-a2ba-7163dd68b5c3" (UID: "9bbff140-4b0c-492c-a2ba-7163dd68b5c3"). InnerVolumeSpecName "kube-api-access-gx24g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:21.802695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:21.802656 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx24g\" (UniqueName: \"kubernetes.io/projected/9bbff140-4b0c-492c-a2ba-7163dd68b5c3-kube-api-access-gx24g\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:22.263413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.263378 5850 generic.go:356] "Generic (PLEG): container finished" podID="9bbff140-4b0c-492c-a2ba-7163dd68b5c3" containerID="c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9" exitCode=0 Nov 02 20:16:22.263682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.263420 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" Nov 02 20:16:22.263682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.263425 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" event={"ID":"9bbff140-4b0c-492c-a2ba-7163dd68b5c3","Type":"ContainerDied","Data":"c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9"} Nov 02 20:16:22.263682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.263568 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c" event={"ID":"9bbff140-4b0c-492c-a2ba-7163dd68b5c3","Type":"ContainerDied","Data":"1612ffe15bee9ad16087d2c5a270b1a9f0c5cf4a38aaf7e0adcca81042417f46"} Nov 02 20:16:22.263682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.263609 5850 scope.go:122] "RemoveContainer" containerID="c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9" Nov 02 20:16:22.273018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.272983 5850 scope.go:122] "RemoveContainer" containerID="c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9" Nov 02 20:16:22.273341 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:22.273319 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9\": container with ID starting with c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9 not found: ID does not exist" containerID="c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9" Nov 02 20:16:22.273397 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.273354 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9"} err="failed to get container status \"c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9\": rpc error: code = NotFound desc = could not find container \"c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9\": container with ID starting with c3d6a752c724ae1ddaf7ba51d1abdff5ee4964f87800a0fc884242f9a01683a9 not found: ID does not exist" Nov 02 20:16:22.434013 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.433978 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c"] Nov 02 20:16:22.435939 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.435912 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19d134-7c5c5f7c79-l7f8c"] Nov 02 20:16:22.511231 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:22.511199 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:23.429916 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:23.429815 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:23.581631 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:23.581566 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:23.823179 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:23.823098 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bbff140-4b0c-492c-a2ba-7163dd68b5c3" path="/var/lib/kubelet/pods/9bbff140-4b0c-492c-a2ba-7163dd68b5c3/volumes" Nov 02 20:16:23.927660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:23.925700 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:24.140419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:24.140358 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:24.748507 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:24.748402 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:25.013006 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:25.012862 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:25.168541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:25.168487 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:25.328987 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:25.328905 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:25.937650 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:25.937618 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:29.301414 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:29.301383 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:30.505802 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:30.505764 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:30.757163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:30.757058 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:30.973861 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:30.973827 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:31.177537 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:31.174647 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:31.752125 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:31.752092 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:32.702331 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:32.702295 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:32.867598 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:32.867570 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:33.017020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:33.008044 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6"] Nov 02 20:16:33.017612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:33.017567 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" containerID="cri-o://e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f" gracePeriod=30 Nov 02 20:16:33.037139 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:33.037102 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:33.180356 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:33.180322 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:33.476749 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:33.476718 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:35.975804 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:35.975769 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:37.721633 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:37.721578 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:37.890913 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:37.890868 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:38.039925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.039895 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c"] Nov 02 20:16:38.040174 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.040151 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" containerID="cri-o://fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871" gracePeriod=30 Nov 02 20:16:38.106530 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.106420 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:38.283769 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.283738 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:16:38.392846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.392819 5850 generic.go:356] "Generic (PLEG): container finished" podID="27ebd778-efab-4d94-b73a-d00738bd2659" containerID="e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f" exitCode=0 Nov 02 20:16:38.393044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.392867 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" Nov 02 20:16:38.393044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.392905 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" event={"ID":"27ebd778-efab-4d94-b73a-d00738bd2659","Type":"ContainerDied","Data":"e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f"} Nov 02 20:16:38.393044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.392972 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6" event={"ID":"27ebd778-efab-4d94-b73a-d00738bd2659","Type":"ContainerDied","Data":"513f528d2f140449866846b4fcb2bee75c44a0066cfd74147f234b9f518c47f6"} Nov 02 20:16:38.393044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.393005 5850 scope.go:122] "RemoveContainer" containerID="e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f" Nov 02 20:16:38.402870 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.402848 5850 scope.go:122] "RemoveContainer" containerID="2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078" Nov 02 20:16:38.413970 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.413938 5850 scope.go:122] "RemoveContainer" containerID="e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f" Nov 02 20:16:38.414341 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:38.414311 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f\": container with ID starting with e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f not found: ID does not exist" containerID="e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f" Nov 02 20:16:38.414541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.414352 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f"} err="failed to get container status \"e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f\": rpc error: code = NotFound desc = could not find container \"e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f\": container with ID starting with e2ff20d1407c0d75b8a7b5948fd49f9a8bfb2bacc4943491528d28b823527d1f not found: ID does not exist" Nov 02 20:16:38.414541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.414373 5850 scope.go:122] "RemoveContainer" containerID="2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078" Nov 02 20:16:38.415369 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:38.415339 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078\": container with ID starting with 2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078 not found: ID does not exist" containerID="2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078" Nov 02 20:16:38.415522 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.415380 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078"} err="failed to get container status \"2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078\": rpc error: code = NotFound desc = could not find container \"2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078\": container with ID starting with 2653d92bcc0f23b8ee0e2f925ebb8c0d536344f66dae3605a89595a5bf593078 not found: ID does not exist" Nov 02 20:16:38.415667 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.415615 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ebd778-efab-4d94-b73a-d00738bd2659-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ebd778-efab-4d94-b73a-d00738bd2659-istiod-ca-cert\") pod \"27ebd778-efab-4d94-b73a-d00738bd2659\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " Nov 02 20:16:38.415667 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.415642 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-certs\") pod \"27ebd778-efab-4d94-b73a-d00738bd2659\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " Nov 02 20:16:38.415790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.415686 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-credential-socket\") pod \"27ebd778-efab-4d94-b73a-d00738bd2659\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " Nov 02 20:16:38.415790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.415715 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27ebd778-efab-4d94-b73a-d00738bd2659-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ebd778-efab-4d94-b73a-d00738bd2659-istio-podinfo\") pod \"27ebd778-efab-4d94-b73a-d00738bd2659\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " Nov 02 20:16:38.415790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.415781 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-istio-token\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-istio-token\") pod \"27ebd778-efab-4d94-b73a-d00738bd2659\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " Nov 02 20:16:38.415950 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.415816 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-socket\") pod \"27ebd778-efab-4d94-b73a-d00738bd2659\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " Nov 02 20:16:38.415950 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.415871 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-data\") pod \"27ebd778-efab-4d94-b73a-d00738bd2659\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " Nov 02 20:16:38.416062 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416022 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-kube-api-access-vzb7z\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-kube-api-access-vzb7z\") pod \"27ebd778-efab-4d94-b73a-d00738bd2659\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " Nov 02 20:16:38.416062 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416028 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ebd778-efab-4d94-b73a-d00738bd2659-istiod-ca-cert" pod "27ebd778-efab-4d94-b73a-d00738bd2659" (UID: "27ebd778-efab-4d94-b73a-d00738bd2659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:38.416062 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416043 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-credential-socket" pod "27ebd778-efab-4d94-b73a-d00738bd2659" (UID: "27ebd778-efab-4d94-b73a-d00738bd2659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:38.416219 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416061 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-envoy\") pod \"27ebd778-efab-4d94-b73a-d00738bd2659\" (UID: \"27ebd778-efab-4d94-b73a-d00738bd2659\") " Nov 02 20:16:38.416219 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416070 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-certs" pod "27ebd778-efab-4d94-b73a-d00738bd2659" (UID: "27ebd778-efab-4d94-b73a-d00738bd2659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:38.416319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416278 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-socket" pod "27ebd778-efab-4d94-b73a-d00738bd2659" (UID: "27ebd778-efab-4d94-b73a-d00738bd2659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:38.416375 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416343 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-data" pod "27ebd778-efab-4d94-b73a-d00738bd2659" (UID: "27ebd778-efab-4d94-b73a-d00738bd2659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:38.416606 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416585 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ebd778-efab-4d94-b73a-d00738bd2659-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.416706 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416611 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.416706 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416626 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.416706 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416640 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.416706 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.416652 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.418783 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.418756 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-kube-api-access-vzb7z" pod "27ebd778-efab-4d94-b73a-d00738bd2659" (UID: "27ebd778-efab-4d94-b73a-d00738bd2659"). InnerVolumeSpecName "kube-api-access-vzb7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:38.418902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.418877 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-istio-token" pod "27ebd778-efab-4d94-b73a-d00738bd2659" (UID: "27ebd778-efab-4d94-b73a-d00738bd2659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:38.419056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.419021 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-envoy" pod "27ebd778-efab-4d94-b73a-d00738bd2659" (UID: "27ebd778-efab-4d94-b73a-d00738bd2659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:38.420708 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.420679 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27ebd778-efab-4d94-b73a-d00738bd2659-istio-podinfo" pod "27ebd778-efab-4d94-b73a-d00738bd2659" (UID: "27ebd778-efab-4d94-b73a-d00738bd2659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:38.421607 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.421585 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf"] Nov 02 20:16:38.517879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.517844 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzb7z\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-kube-api-access-vzb7z\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.517879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.517876 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ebd778-efab-4d94-b73a-d00738bd2659-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.517879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.517885 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ebd778-efab-4d94-b73a-d00738bd2659-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.518132 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.517894 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ebd778-efab-4d94-b73a-d00738bd2659-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.536867 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.536828 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:38.817845 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.817806 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6"] Nov 02 20:16:38.822041 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:38.822010 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa24-openshift-default-59f48ffbc-zl9d6"] Nov 02 20:16:39.106833 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:39.106740 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:39.268257 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:39.268219 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:39.815582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:39.815551 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" path="/var/lib/kubelet/pods/27ebd778-efab-4d94-b73a-d00738bd2659/volumes" Nov 02 20:16:40.773504 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:40.773441 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:42.471548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:42.471508 5850 patch_prober.go:29] interesting pod/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:16:42.471962 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:42.471577 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:16:43.300169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.300139 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:16:43.366883 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.366855 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-data\") pod \"50407317-673e-4dfa-9ee2-b70a95b97e31\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " Nov 02 20:16:43.367044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.366898 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-socket\") pod \"50407317-673e-4dfa-9ee2-b70a95b97e31\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " Nov 02 20:16:43.367044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.366923 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-credential-socket\") pod \"50407317-673e-4dfa-9ee2-b70a95b97e31\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " Nov 02 20:16:43.367044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.366955 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-token\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-token\") pod \"50407317-673e-4dfa-9ee2-b70a95b97e31\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " Nov 02 20:16:43.367044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.366978 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50407317-673e-4dfa-9ee2-b70a95b97e31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50407317-673e-4dfa-9ee2-b70a95b97e31-istiod-ca-cert\") pod \"50407317-673e-4dfa-9ee2-b70a95b97e31\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " Nov 02 20:16:43.367044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367009 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-kube-api-access-rgm2h\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-kube-api-access-rgm2h\") pod \"50407317-673e-4dfa-9ee2-b70a95b97e31\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " Nov 02 20:16:43.367339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367058 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-envoy\") pod \"50407317-673e-4dfa-9ee2-b70a95b97e31\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " Nov 02 20:16:43.367339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367183 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-data" pod "50407317-673e-4dfa-9ee2-b70a95b97e31" (UID: "50407317-673e-4dfa-9ee2-b70a95b97e31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:43.367339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367199 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-socket" pod "50407317-673e-4dfa-9ee2-b70a95b97e31" (UID: "50407317-673e-4dfa-9ee2-b70a95b97e31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:43.367339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367277 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-credential-socket" pod "50407317-673e-4dfa-9ee2-b70a95b97e31" (UID: "50407317-673e-4dfa-9ee2-b70a95b97e31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:43.367339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367278 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-podinfo\") pod \"50407317-673e-4dfa-9ee2-b70a95b97e31\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " Nov 02 20:16:43.367640 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367412 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-certs\") pod \"50407317-673e-4dfa-9ee2-b70a95b97e31\" (UID: \"50407317-673e-4dfa-9ee2-b70a95b97e31\") " Nov 02 20:16:43.367640 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367530 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50407317-673e-4dfa-9ee2-b70a95b97e31-istiod-ca-cert" pod "50407317-673e-4dfa-9ee2-b70a95b97e31" (UID: "50407317-673e-4dfa-9ee2-b70a95b97e31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:43.367747 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367732 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-certs" pod "50407317-673e-4dfa-9ee2-b70a95b97e31" (UID: "50407317-673e-4dfa-9ee2-b70a95b97e31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:43.367909 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367888 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.367909 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367910 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50407317-673e-4dfa-9ee2-b70a95b97e31-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.368073 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367924 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.368073 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367937 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.368073 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.367949 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.370121 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.370053 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-kube-api-access-rgm2h" pod "50407317-673e-4dfa-9ee2-b70a95b97e31" (UID: "50407317-673e-4dfa-9ee2-b70a95b97e31"). InnerVolumeSpecName "kube-api-access-rgm2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:43.372917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.372884 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-envoy" pod "50407317-673e-4dfa-9ee2-b70a95b97e31" (UID: "50407317-673e-4dfa-9ee2-b70a95b97e31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:43.372917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.372913 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-token" pod "50407317-673e-4dfa-9ee2-b70a95b97e31" (UID: "50407317-673e-4dfa-9ee2-b70a95b97e31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:43.373082 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.372979 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-podinfo" pod "50407317-673e-4dfa-9ee2-b70a95b97e31" (UID: "50407317-673e-4dfa-9ee2-b70a95b97e31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:43.426819 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.426782 5850 generic.go:356] "Generic (PLEG): container finished" podID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerID="fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871" exitCode=0 Nov 02 20:16:43.427033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.426840 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" Nov 02 20:16:43.427033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.426851 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" event={"ID":"50407317-673e-4dfa-9ee2-b70a95b97e31","Type":"ContainerDied","Data":"fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871"} Nov 02 20:16:43.427033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.426915 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c" event={"ID":"50407317-673e-4dfa-9ee2-b70a95b97e31","Type":"ContainerDied","Data":"a7bee769722b52d2dbab0fd16d4937fab7cd493681451e68a4b7302747fd74c5"} Nov 02 20:16:43.427033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.426946 5850 scope.go:122] "RemoveContainer" containerID="fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871" Nov 02 20:16:43.439057 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.439021 5850 scope.go:122] "RemoveContainer" containerID="401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598" Nov 02 20:16:43.448498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.448462 5850 scope.go:122] "RemoveContainer" containerID="fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871" Nov 02 20:16:43.448790 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:43.448768 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871\": container with ID starting with fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871 not found: ID does not exist" containerID="fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871" Nov 02 20:16:43.448869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.448801 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871"} err="failed to get container status \"fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871\": rpc error: code = NotFound desc = could not find container \"fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871\": container with ID starting with fea208bc3ce57d9a0a1d82321414e08c494130228885fb23850ea9a651165871 not found: ID does not exist" Nov 02 20:16:43.448869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.448823 5850 scope.go:122] "RemoveContainer" containerID="401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598" Nov 02 20:16:43.449110 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:43.449091 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598\": container with ID starting with 401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598 not found: ID does not exist" containerID="401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598" Nov 02 20:16:43.449167 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.449119 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598"} err="failed to get container status \"401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598\": rpc error: code = NotFound desc = could not find container \"401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598\": container with ID starting with 401481dc78de05dd25eae48040c0d1ddce0f61ce2cd8b593dd8851d0289a0598 not found: ID does not exist" Nov 02 20:16:43.468911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.468869 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.468911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.468905 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.468911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.468916 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.469165 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.468926 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgm2h\" (UniqueName: \"kubernetes.io/projected/50407317-673e-4dfa-9ee2-b70a95b97e31-kube-api-access-rgm2h\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.585385 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.585250 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c"] Nov 02 20:16:43.592607 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.592573 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-19d134-openshift-default-7cd87d6b5-v9v4c"] Nov 02 20:16:43.661772 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.661689 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:43.814546 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.814505 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:43.816736 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:43.815728 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" path="/var/lib/kubelet/pods/50407317-673e-4dfa-9ee2-b70a95b97e31/volumes" Nov 02 20:16:44.761942 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:44.761904 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:16:45.435198 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:45.435164 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:46.266159 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:46.266128 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:46.413489 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:46.413453 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:49.357823 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.357784 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl"] Nov 02 20:16:49.358203 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.358021 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" podUID="519e3ebf-d09c-45b5-a773-42b1dad0c314" containerName="maas-api" containerID="cri-o://2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739" gracePeriod=30 Nov 02 20:16:49.478673 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.478642 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:49.641131 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.641102 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 20:16:49.775710 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.775673 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls\") pod \"519e3ebf-d09c-45b5-a773-42b1dad0c314\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " Nov 02 20:16:49.775901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.775767 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-metrics-certs\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-metrics-certs\") pod \"519e3ebf-d09c-45b5-a773-42b1dad0c314\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " Nov 02 20:16:49.775983 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.775918 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519e3ebf-d09c-45b5-a773-42b1dad0c314-kube-api-access-69vtj\" (UniqueName: \"kubernetes.io/projected/519e3ebf-d09c-45b5-a773-42b1dad0c314-kube-api-access-69vtj\") pod \"519e3ebf-d09c-45b5-a773-42b1dad0c314\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " Nov 02 20:16:49.776131 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.776085 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519e3ebf-d09c-45b5-a773-42b1dad0c314-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519e3ebf-d09c-45b5-a773-42b1dad0c314-openshift-service-ca\") pod \"519e3ebf-d09c-45b5-a773-42b1dad0c314\" (UID: \"519e3ebf-d09c-45b5-a773-42b1dad0c314\") " Nov 02 20:16:49.776681 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.776526 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519e3ebf-d09c-45b5-a773-42b1dad0c314-openshift-service-ca" pod "519e3ebf-d09c-45b5-a773-42b1dad0c314" (UID: "519e3ebf-d09c-45b5-a773-42b1dad0c314"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:49.776681 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.776653 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519e3ebf-d09c-45b5-a773-42b1dad0c314-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:49.777917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.777891 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls" pod "519e3ebf-d09c-45b5-a773-42b1dad0c314" (UID: "519e3ebf-d09c-45b5-a773-42b1dad0c314"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:49.778379 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.778356 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-metrics-certs" pod "519e3ebf-d09c-45b5-a773-42b1dad0c314" (UID: "519e3ebf-d09c-45b5-a773-42b1dad0c314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:49.778454 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.778391 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519e3ebf-d09c-45b5-a773-42b1dad0c314-kube-api-access-69vtj" pod "519e3ebf-d09c-45b5-a773-42b1dad0c314" (UID: "519e3ebf-d09c-45b5-a773-42b1dad0c314"). InnerVolumeSpecName "kube-api-access-69vtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:49.880434 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.880217 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:49.880434 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.880262 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69vtj\" (UniqueName: \"kubernetes.io/projected/519e3ebf-d09c-45b5-a773-42b1dad0c314-kube-api-access-69vtj\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:49.880434 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:49.880277 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519e3ebf-d09c-45b5-a773-42b1dad0c314-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:50.468409 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.468373 5850 generic.go:356] "Generic (PLEG): container finished" podID="519e3ebf-d09c-45b5-a773-42b1dad0c314" containerID="2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739" exitCode=0 Nov 02 20:16:50.468873 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.468426 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" Nov 02 20:16:50.468873 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.468463 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" event={"ID":"519e3ebf-d09c-45b5-a773-42b1dad0c314","Type":"ContainerDied","Data":"2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739"} Nov 02 20:16:50.468873 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.468545 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl" event={"ID":"519e3ebf-d09c-45b5-a773-42b1dad0c314","Type":"ContainerDied","Data":"7ba9d00485eca54a66f3b51d4b6d27a6c7bda595bf226091e00a0bd38f47a966"} Nov 02 20:16:50.468873 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.468568 5850 scope.go:122] "RemoveContainer" containerID="2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739" Nov 02 20:16:50.478426 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.478405 5850 scope.go:122] "RemoveContainer" containerID="2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739" Nov 02 20:16:50.478798 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:50.478774 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739\": container with ID starting with 2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739 not found: ID does not exist" containerID="2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739" Nov 02 20:16:50.478866 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.478807 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739"} err="failed to get container status \"2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739\": rpc error: code = NotFound desc = could not find container \"2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739\": container with ID starting with 2115506e983d2a4c3fadbb034bbe84613a4a79cc321763f376207b7305cc8739 not found: ID does not exist" Nov 02 20:16:50.639073 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.639032 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl"] Nov 02 20:16:50.646047 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.645954 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5c46af-7497444c9-5fpnl"] Nov 02 20:16:50.825713 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.825628 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:50.984089 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:50.984050 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:51.148003 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:51.147969 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:51.314791 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:51.314754 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:51.814843 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:51.814807 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519e3ebf-d09c-45b5-a773-42b1dad0c314" path="/var/lib/kubelet/pods/519e3ebf-d09c-45b5-a773-42b1dad0c314/volumes" Nov 02 20:16:51.825655 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:51.825626 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:53.442614 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.442564 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" containerID="cri-o://ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00" gracePeriod=45 Nov 02 20:16:53.707243 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.707180 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 20:16:53.821563 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.821531 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-home\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-home\") pod \"2dc565ff-fda0-4743-add0-891392bc36dc\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " Nov 02 20:16:53.821735 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.821638 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-tmp-dir\") pod \"2dc565ff-fda0-4743-add0-891392bc36dc\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " Nov 02 20:16:53.821735 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.821687 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-dshm\") pod \"2dc565ff-fda0-4743-add0-891392bc36dc\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " Nov 02 20:16:53.821869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.821758 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc565ff-fda0-4743-add0-891392bc36dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc565ff-fda0-4743-add0-891392bc36dc-tls-certs\") pod \"2dc565ff-fda0-4743-add0-891392bc36dc\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " Nov 02 20:16:53.821869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.821795 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc565ff-fda0-4743-add0-891392bc36dc-kube-api-access-gv9lx\" (UniqueName: \"kubernetes.io/projected/2dc565ff-fda0-4743-add0-891392bc36dc-kube-api-access-gv9lx\") pod \"2dc565ff-fda0-4743-add0-891392bc36dc\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " Nov 02 20:16:53.821869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.821835 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-model-cache\") pod \"2dc565ff-fda0-4743-add0-891392bc36dc\" (UID: \"2dc565ff-fda0-4743-add0-891392bc36dc\") " Nov 02 20:16:53.822007 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.821842 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-home" pod "2dc565ff-fda0-4743-add0-891392bc36dc" (UID: "2dc565ff-fda0-4743-add0-891392bc36dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.822007 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.821926 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-tmp-dir" pod "2dc565ff-fda0-4743-add0-891392bc36dc" (UID: "2dc565ff-fda0-4743-add0-891392bc36dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.822209 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.822190 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-model-cache" pod "2dc565ff-fda0-4743-add0-891392bc36dc" (UID: "2dc565ff-fda0-4743-add0-891392bc36dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.822288 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.822268 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.822334 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.822295 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.822375 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.822331 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.824247 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.824175 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-dshm" pod "2dc565ff-fda0-4743-add0-891392bc36dc" (UID: "2dc565ff-fda0-4743-add0-891392bc36dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.824396 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.824313 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc565ff-fda0-4743-add0-891392bc36dc-kube-api-access-gv9lx" pod "2dc565ff-fda0-4743-add0-891392bc36dc" (UID: "2dc565ff-fda0-4743-add0-891392bc36dc"). InnerVolumeSpecName "kube-api-access-gv9lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:53.824569 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.824548 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc565ff-fda0-4743-add0-891392bc36dc-tls-certs" pod "2dc565ff-fda0-4743-add0-891392bc36dc" (UID: "2dc565ff-fda0-4743-add0-891392bc36dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:53.923169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.923129 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc565ff-fda0-4743-add0-891392bc36dc-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.923169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.923158 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc565ff-fda0-4743-add0-891392bc36dc-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.923169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:53.923173 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9lx\" (UniqueName: \"kubernetes.io/projected/2dc565ff-fda0-4743-add0-891392bc36dc-kube-api-access-gv9lx\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:54.492157 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:54.492118 5850 generic.go:356] "Generic (PLEG): container finished" podID="2dc565ff-fda0-4743-add0-891392bc36dc" containerID="ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00" exitCode=0 Nov 02 20:16:54.492691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:54.492189 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" Nov 02 20:16:54.492691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:54.492209 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" event={"ID":"2dc565ff-fda0-4743-add0-891392bc36dc","Type":"ContainerDied","Data":"ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00"} Nov 02 20:16:54.492691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:54.492258 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf" event={"ID":"2dc565ff-fda0-4743-add0-891392bc36dc","Type":"ContainerDied","Data":"932f1b674a4877ebb0200823b0fac62ad11f329fef659af104ca250b625c5c3a"} Nov 02 20:16:54.492691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:54.492275 5850 scope.go:122] "RemoveContainer" containerID="ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00" Nov 02 20:16:54.502116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:54.502095 5850 scope.go:122] "RemoveContainer" containerID="ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00" Nov 02 20:16:54.502386 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:16:54.502365 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00\": container with ID starting with ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00 not found: ID does not exist" containerID="ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00" Nov 02 20:16:54.502428 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:54.502397 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00"} err="failed to get container status \"ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00\": rpc error: code = NotFound desc = could not find container \"ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00\": container with ID starting with ac18429ff16abd4b4beafa8c36ab0afe90afa105d45952aaf77bb6be9541ea00 not found: ID does not exist" Nov 02 20:16:54.659437 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:54.659406 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf"] Nov 02 20:16:54.661519 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:54.661499 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-w0-43ceaa-kserve-b7b75d898-nzjqf"] Nov 02 20:16:55.748917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:55.748884 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:55.816727 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:55.816687 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" path="/var/lib/kubelet/pods/2dc565ff-fda0-4743-add0-891392bc36dc/volumes" Nov 02 20:16:57.621897 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:57.621857 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:57.819923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:57.819885 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:57.972455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:16:57.972363 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:00.136956 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:00.136921 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:04.734869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:04.734831 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:08.428093 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.428052 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p"] Nov 02 20:17:08.428561 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.428267 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" podUID="ddfcd7fe-218c-4efd-ba81-45f03d929f4b" containerName="maas-api" containerID="cri-o://fd047667a6f3f31c51c53419f093d50ecd35d9ef76f94f7d9669b72e5736955d" gracePeriod=30 Nov 02 20:17:08.544707 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.544669 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:08.571959 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.571924 5850 generic.go:356] "Generic (PLEG): container finished" podID="ddfcd7fe-218c-4efd-ba81-45f03d929f4b" containerID="fd047667a6f3f31c51c53419f093d50ecd35d9ef76f94f7d9669b72e5736955d" exitCode=0 Nov 02 20:17:08.572134 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.571999 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" event={"ID":"ddfcd7fe-218c-4efd-ba81-45f03d929f4b","Type":"ContainerDied","Data":"fd047667a6f3f31c51c53419f093d50ecd35d9ef76f94f7d9669b72e5736955d"} Nov 02 20:17:08.697887 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.697861 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:17:08.795104 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.795060 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-kube-api-access-hpd6b\" (UniqueName: \"kubernetes.io/projected/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-kube-api-access-hpd6b\") pod \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " Nov 02 20:17:08.795285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.795210 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-openshift-service-ca\") pod \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " Nov 02 20:17:08.795285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.795261 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-metrics-certs\") pod \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " Nov 02 20:17:08.795462 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.795305 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-maas-api-tls\") pod \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\" (UID: \"ddfcd7fe-218c-4efd-ba81-45f03d929f4b\") " Nov 02 20:17:08.796900 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.796591 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-openshift-service-ca" pod "ddfcd7fe-218c-4efd-ba81-45f03d929f4b" (UID: "ddfcd7fe-218c-4efd-ba81-45f03d929f4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:08.800708 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.800678 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-kube-api-access-hpd6b" pod "ddfcd7fe-218c-4efd-ba81-45f03d929f4b" (UID: "ddfcd7fe-218c-4efd-ba81-45f03d929f4b"). InnerVolumeSpecName "kube-api-access-hpd6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:08.805380 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.805338 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-metrics-certs" pod "ddfcd7fe-218c-4efd-ba81-45f03d929f4b" (UID: "ddfcd7fe-218c-4efd-ba81-45f03d929f4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:08.806162 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.806112 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-maas-api-tls" pod "ddfcd7fe-218c-4efd-ba81-45f03d929f4b" (UID: "ddfcd7fe-218c-4efd-ba81-45f03d929f4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:08.896988 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.896937 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:08.896988 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.896979 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:08.896988 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.896992 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:08.897256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:08.897005 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpd6b\" (UniqueName: \"kubernetes.io/projected/ddfcd7fe-218c-4efd-ba81-45f03d929f4b-kube-api-access-hpd6b\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:09.579688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:09.579607 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" event={"ID":"ddfcd7fe-218c-4efd-ba81-45f03d929f4b","Type":"ContainerDied","Data":"79216209370b46ff4c640573f01c62ee38b809839ea21124f9920ae9752f101f"} Nov 02 20:17:09.579688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:09.579629 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p" Nov 02 20:17:09.579688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:09.579672 5850 scope.go:122] "RemoveContainer" containerID="fd047667a6f3f31c51c53419f093d50ecd35d9ef76f94f7d9669b72e5736955d" Nov 02 20:17:09.747287 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:09.747254 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p"] Nov 02 20:17:09.748983 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:09.748956 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-13e739-594cd47dc5-9dx9p"] Nov 02 20:17:09.814366 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:09.814333 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfcd7fe-218c-4efd-ba81-45f03d929f4b" path="/var/lib/kubelet/pods/ddfcd7fe-218c-4efd-ba81-45f03d929f4b/volumes" Nov 02 20:17:09.935080 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:09.935045 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:10.132097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:10.132064 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:10.314079 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:10.313991 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:10.796593 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:10.796557 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:14.169229 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:14.169193 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:14.587592 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:14.587504 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:14.761671 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:14.761633 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:17:14.767452 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:14.767426 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:15.514837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:15.514803 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:15.675986 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:15.675953 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:15.978259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:15.978220 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:16.143374 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:16.143340 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:16.309878 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:16.309793 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:16.517443 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:16.517412 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:17.758170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:17.758126 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:17.943329 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:17.943294 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:18.062502 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:18.062393 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7"] Nov 02 20:17:18.062694 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:18.062628 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" containerID="cri-o://1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb" gracePeriod=30 Nov 02 20:17:18.103348 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:18.103318 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:19.519534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:19.519464 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:20.706900 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:20.706870 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:22.409833 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:22.409794 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:22.568001 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:22.567960 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:22.801428 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:22.801335 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv"] Nov 02 20:17:22.801692 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:22.801668 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" containerID="cri-o://556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6" gracePeriod=30 Nov 02 20:17:22.822071 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:22.822038 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:23.321794 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.321765 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:17:23.479164 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479128 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-data\") pod \"c7cf89b2-daf1-44e0-a445-49d67c92861c\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " Nov 02 20:17:23.479164 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479169 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-token\") pod \"c7cf89b2-daf1-44e0-a445-49d67c92861c\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479194 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-podinfo\") pod \"c7cf89b2-daf1-44e0-a445-49d67c92861c\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479263 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-envoy\") pod \"c7cf89b2-daf1-44e0-a445-49d67c92861c\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479337 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-credential-socket\") pod \"c7cf89b2-daf1-44e0-a445-49d67c92861c\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479406 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-kube-api-access-sx2gb\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-kube-api-access-sx2gb\") pod \"c7cf89b2-daf1-44e0-a445-49d67c92861c\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479440 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-data" pod "c7cf89b2-daf1-44e0-a445-49d67c92861c" (UID: "c7cf89b2-daf1-44e0-a445-49d67c92861c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479446 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-certs\") pod \"c7cf89b2-daf1-44e0-a445-49d67c92861c\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479620 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7cf89b2-daf1-44e0-a445-49d67c92861c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf89b2-daf1-44e0-a445-49d67c92861c-istiod-ca-cert\") pod \"c7cf89b2-daf1-44e0-a445-49d67c92861c\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479678 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-socket\") pod \"c7cf89b2-daf1-44e0-a445-49d67c92861c\" (UID: \"c7cf89b2-daf1-44e0-a445-49d67c92861c\") " Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479728 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-certs" pod "c7cf89b2-daf1-44e0-a445-49d67c92861c" (UID: "c7cf89b2-daf1-44e0-a445-49d67c92861c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:23.479795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479758 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-credential-socket" pod "c7cf89b2-daf1-44e0-a445-49d67c92861c" (UID: "c7cf89b2-daf1-44e0-a445-49d67c92861c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:23.480337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.480026 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-socket" pod "c7cf89b2-daf1-44e0-a445-49d67c92861c" (UID: "c7cf89b2-daf1-44e0-a445-49d67c92861c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:23.480337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.479962 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7cf89b2-daf1-44e0-a445-49d67c92861c-istiod-ca-cert" pod "c7cf89b2-daf1-44e0-a445-49d67c92861c" (UID: "c7cf89b2-daf1-44e0-a445-49d67c92861c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:23.480337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.480160 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:23.480337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.480184 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:23.480337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.480199 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:23.480337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.480216 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:23.481706 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.481684 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-podinfo" pod "c7cf89b2-daf1-44e0-a445-49d67c92861c" (UID: "c7cf89b2-daf1-44e0-a445-49d67c92861c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:17:23.481835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.481705 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-envoy" pod "c7cf89b2-daf1-44e0-a445-49d67c92861c" (UID: "c7cf89b2-daf1-44e0-a445-49d67c92861c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:23.481835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.481727 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-token" pod "c7cf89b2-daf1-44e0-a445-49d67c92861c" (UID: "c7cf89b2-daf1-44e0-a445-49d67c92861c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:23.481835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.481774 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-kube-api-access-sx2gb" pod "c7cf89b2-daf1-44e0-a445-49d67c92861c" (UID: "c7cf89b2-daf1-44e0-a445-49d67c92861c"). InnerVolumeSpecName "kube-api-access-sx2gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:23.581054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.581021 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf89b2-daf1-44e0-a445-49d67c92861c-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:23.581054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.581053 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:23.581054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.581063 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:23.581306 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.581073 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf89b2-daf1-44e0-a445-49d67c92861c-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:23.581306 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.581082 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx2gb\" (UniqueName: \"kubernetes.io/projected/c7cf89b2-daf1-44e0-a445-49d67c92861c-kube-api-access-sx2gb\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:23.665941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.665909 5850 generic.go:356] "Generic (PLEG): container finished" podID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerID="1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb" exitCode=0 Nov 02 20:17:23.666116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.665949 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" event={"ID":"c7cf89b2-daf1-44e0-a445-49d67c92861c","Type":"ContainerDied","Data":"1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb"} Nov 02 20:17:23.666116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.665961 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" Nov 02 20:17:23.666116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.665998 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7" event={"ID":"c7cf89b2-daf1-44e0-a445-49d67c92861c","Type":"ContainerDied","Data":"f3b17f7dcdce12f9109bafa6df90fbd9012c712c51f00d38c2b7d5e532321f05"} Nov 02 20:17:23.666116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.666020 5850 scope.go:122] "RemoveContainer" containerID="1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb" Nov 02 20:17:23.676249 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.676231 5850 scope.go:122] "RemoveContainer" containerID="8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197" Nov 02 20:17:23.687412 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.687384 5850 scope.go:122] "RemoveContainer" containerID="1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb" Nov 02 20:17:23.687847 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:23.687817 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb\": container with ID starting with 1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb not found: ID does not exist" containerID="1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb" Nov 02 20:17:23.687950 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.687857 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb"} err="failed to get container status \"1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb\": rpc error: code = NotFound desc = could not find container \"1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb\": container with ID starting with 1c3996843fafd917c3511c9630d596df314416b328d280c853f2d85ef2445acb not found: ID does not exist" Nov 02 20:17:23.687950 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.687876 5850 scope.go:122] "RemoveContainer" containerID="8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197" Nov 02 20:17:23.688193 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:23.688175 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197\": container with ID starting with 8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197 not found: ID does not exist" containerID="8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197" Nov 02 20:17:23.688240 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.688199 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197"} err="failed to get container status \"8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197\": rpc error: code = NotFound desc = could not find container \"8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197\": container with ID starting with 8f2567b218aab2a41d6a1f023875a403ad5107ce1295dbee0c2340def0fff197 not found: ID does not exist" Nov 02 20:17:23.834186 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.834096 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7"] Nov 02 20:17:23.837906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:23.837874 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-13e739-openshift-default-6b6f585565-zcgl7"] Nov 02 20:17:25.359853 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:25.359820 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:25.815040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:25.815006 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" path="/var/lib/kubelet/pods/c7cf89b2-daf1-44e0-a445-49d67c92861c/volumes" Nov 02 20:17:26.373869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:26.373835 5850 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:17:26.374293 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:26.373903 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:17:28.057708 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.057679 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 20:17:28.234906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.234867 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-podinfo\") pod \"71cb4481-97cd-44e3-a14a-01dbcac2752c\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " Nov 02 20:17:28.235111 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235006 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-data\") pod \"71cb4481-97cd-44e3-a14a-01dbcac2752c\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " Nov 02 20:17:28.235111 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235045 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-kube-api-access-8vfwh\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-kube-api-access-8vfwh\") pod \"71cb4481-97cd-44e3-a14a-01dbcac2752c\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " Nov 02 20:17:28.235111 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235093 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71cb4481-97cd-44e3-a14a-01dbcac2752c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb4481-97cd-44e3-a14a-01dbcac2752c-istiod-ca-cert\") pod \"71cb4481-97cd-44e3-a14a-01dbcac2752c\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " Nov 02 20:17:28.235299 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235121 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-token\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-token\") pod \"71cb4481-97cd-44e3-a14a-01dbcac2752c\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " Nov 02 20:17:28.235299 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235156 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-envoy\") pod \"71cb4481-97cd-44e3-a14a-01dbcac2752c\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " Nov 02 20:17:28.235299 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235206 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-credential-socket\") pod \"71cb4481-97cd-44e3-a14a-01dbcac2752c\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " Nov 02 20:17:28.235299 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235274 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-socket\") pod \"71cb4481-97cd-44e3-a14a-01dbcac2752c\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " Nov 02 20:17:28.235447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235309 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-certs\") pod \"71cb4481-97cd-44e3-a14a-01dbcac2752c\" (UID: \"71cb4481-97cd-44e3-a14a-01dbcac2752c\") " Nov 02 20:17:28.235447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235409 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-data" pod "71cb4481-97cd-44e3-a14a-01dbcac2752c" (UID: "71cb4481-97cd-44e3-a14a-01dbcac2752c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:28.235773 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235735 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-socket" pod "71cb4481-97cd-44e3-a14a-01dbcac2752c" (UID: "71cb4481-97cd-44e3-a14a-01dbcac2752c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:28.235952 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.235803 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:28.236043 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.236019 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-certs" pod "71cb4481-97cd-44e3-a14a-01dbcac2752c" (UID: "71cb4481-97cd-44e3-a14a-01dbcac2752c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:28.236105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.236037 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cb4481-97cd-44e3-a14a-01dbcac2752c-istiod-ca-cert" pod "71cb4481-97cd-44e3-a14a-01dbcac2752c" (UID: "71cb4481-97cd-44e3-a14a-01dbcac2752c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:28.236286 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.236254 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-credential-socket" pod "71cb4481-97cd-44e3-a14a-01dbcac2752c" (UID: "71cb4481-97cd-44e3-a14a-01dbcac2752c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:28.237855 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.237824 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-podinfo" pod "71cb4481-97cd-44e3-a14a-01dbcac2752c" (UID: "71cb4481-97cd-44e3-a14a-01dbcac2752c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:17:28.238152 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.238131 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-envoy" pod "71cb4481-97cd-44e3-a14a-01dbcac2752c" (UID: "71cb4481-97cd-44e3-a14a-01dbcac2752c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:28.238152 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.238140 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-token" pod "71cb4481-97cd-44e3-a14a-01dbcac2752c" (UID: "71cb4481-97cd-44e3-a14a-01dbcac2752c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:28.238425 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.238406 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-kube-api-access-8vfwh" pod "71cb4481-97cd-44e3-a14a-01dbcac2752c" (UID: "71cb4481-97cd-44e3-a14a-01dbcac2752c"). InnerVolumeSpecName "kube-api-access-8vfwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:28.337033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.336989 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:28.337033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.337026 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:28.337033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.337037 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:28.337285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.337046 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:28.337285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.337056 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vfwh\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-kube-api-access-8vfwh\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:28.337285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.337067 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cb4481-97cd-44e3-a14a-01dbcac2752c-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:28.337285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.337075 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:28.337285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.337084 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cb4481-97cd-44e3-a14a-01dbcac2752c-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:28.697558 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.697522 5850 generic.go:356] "Generic (PLEG): container finished" podID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerID="556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6" exitCode=0 Nov 02 20:17:28.697743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.697574 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" Nov 02 20:17:28.697743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.697576 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" event={"ID":"71cb4481-97cd-44e3-a14a-01dbcac2752c","Type":"ContainerDied","Data":"556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6"} Nov 02 20:17:28.697743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.697693 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv" event={"ID":"71cb4481-97cd-44e3-a14a-01dbcac2752c","Type":"ContainerDied","Data":"af93fd83df8b8e2b8c88d712b0fe99e20c3bbb629450565f8248d7c011264189"} Nov 02 20:17:28.697743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.697713 5850 scope.go:122] "RemoveContainer" containerID="556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6" Nov 02 20:17:28.708055 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.708037 5850 scope.go:122] "RemoveContainer" containerID="7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54" Nov 02 20:17:28.717277 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.717258 5850 scope.go:122] "RemoveContainer" containerID="556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6" Nov 02 20:17:28.717564 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:28.717543 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6\": container with ID starting with 556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6 not found: ID does not exist" containerID="556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6" Nov 02 20:17:28.717609 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.717574 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6"} err="failed to get container status \"556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6\": rpc error: code = NotFound desc = could not find container \"556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6\": container with ID starting with 556b72942500031d9ba060e2942b5df622f9ebf570ae15864d94f46624a04df6 not found: ID does not exist" Nov 02 20:17:28.717609 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.717591 5850 scope.go:122] "RemoveContainer" containerID="7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54" Nov 02 20:17:28.717856 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:28.717837 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54\": container with ID starting with 7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54 not found: ID does not exist" containerID="7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54" Nov 02 20:17:28.717914 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.717866 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54"} err="failed to get container status \"7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54\": rpc error: code = NotFound desc = could not find container \"7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54\": container with ID starting with 7f7aad61004fb0208ebd6080305e100962eba2f398ac1adf0d9229368d3fdf54 not found: ID does not exist" Nov 02 20:17:28.864590 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.864551 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv"] Nov 02 20:17:28.916042 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:28.916005 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d1aaed-openshift-default-74f7c4ff-mnzmv"] Nov 02 20:17:29.814971 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:29.814934 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" path="/var/lib/kubelet/pods/71cb4481-97cd-44e3-a14a-01dbcac2752c/volumes" Nov 02 20:17:30.166170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:30.166140 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:30.405807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:30.405765 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk"] Nov 02 20:17:30.522772 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:30.522682 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649"] Nov 02 20:17:30.817624 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:30.817546 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:30.962608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:30.962572 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:34.726947 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:34.726911 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:35.406373 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.406325 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" podUID="e80659f4-faea-4edc-81e9-e525fed8afeb" containerName="payload-processing" containerID="cri-o://d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789" gracePeriod=25 Nov 02 20:17:35.516726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.516694 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:35.523552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.523516 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" podUID="6e76b596-bf27-4ec8-b864-8eb365f150fc" containerName="payload-pre-processing" containerID="cri-o://a8f064fe844b287a9bdfd040cb3060f6a2f90315cfab660310c0af90d1426c14" gracePeriod=25 Nov 02 20:17:35.690461 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.690418 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 20:17:35.719985 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.719065 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80659f4-faea-4edc-81e9-e525fed8afeb-kube-api-access-ghmpp\" (UniqueName: \"kubernetes.io/projected/e80659f4-faea-4edc-81e9-e525fed8afeb-kube-api-access-ghmpp\") pod \"e80659f4-faea-4edc-81e9-e525fed8afeb\" (UID: \"e80659f4-faea-4edc-81e9-e525fed8afeb\") " Nov 02 20:17:35.719985 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.719219 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80659f4-faea-4edc-81e9-e525fed8afeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80659f4-faea-4edc-81e9-e525fed8afeb-plugins-config-volume\") pod \"e80659f4-faea-4edc-81e9-e525fed8afeb\" (UID: \"e80659f4-faea-4edc-81e9-e525fed8afeb\") " Nov 02 20:17:35.720439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.720405 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80659f4-faea-4edc-81e9-e525fed8afeb-plugins-config-volume" pod "e80659f4-faea-4edc-81e9-e525fed8afeb" (UID: "e80659f4-faea-4edc-81e9-e525fed8afeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:35.721206 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.721182 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:35.723926 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.723883 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80659f4-faea-4edc-81e9-e525fed8afeb-kube-api-access-ghmpp" pod "e80659f4-faea-4edc-81e9-e525fed8afeb" (UID: "e80659f4-faea-4edc-81e9-e525fed8afeb"). InnerVolumeSpecName "kube-api-access-ghmpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:35.737126 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.737020 5850 generic.go:356] "Generic (PLEG): container finished" podID="6e76b596-bf27-4ec8-b864-8eb365f150fc" containerID="a8f064fe844b287a9bdfd040cb3060f6a2f90315cfab660310c0af90d1426c14" exitCode=0 Nov 02 20:17:35.737126 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.737095 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" event={"ID":"6e76b596-bf27-4ec8-b864-8eb365f150fc","Type":"ContainerDied","Data":"a8f064fe844b287a9bdfd040cb3060f6a2f90315cfab660310c0af90d1426c14"} Nov 02 20:17:35.738851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.738825 5850 generic.go:356] "Generic (PLEG): container finished" podID="e80659f4-faea-4edc-81e9-e525fed8afeb" containerID="d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789" exitCode=0 Nov 02 20:17:35.738984 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.738875 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" Nov 02 20:17:35.738984 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.738920 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" event={"ID":"e80659f4-faea-4edc-81e9-e525fed8afeb","Type":"ContainerDied","Data":"d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789"} Nov 02 20:17:35.739140 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.738988 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk" event={"ID":"e80659f4-faea-4edc-81e9-e525fed8afeb","Type":"ContainerDied","Data":"d8fc316ea3851987222c9b808227ff0c470312190daa4d80533e37748eba335e"} Nov 02 20:17:35.739140 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.739024 5850 scope.go:122] "RemoveContainer" containerID="d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789" Nov 02 20:17:35.749366 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.749346 5850 scope.go:122] "RemoveContainer" containerID="d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789" Nov 02 20:17:35.749855 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:35.749828 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789\": container with ID starting with d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789 not found: ID does not exist" containerID="d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789" Nov 02 20:17:35.749966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.749863 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789"} err="failed to get container status \"d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789\": rpc error: code = NotFound desc = could not find container \"d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789\": container with ID starting with d189941d06ac7bd9814692773d67f7cfe649043455b445d14f3de84cb466e789 not found: ID does not exist" Nov 02 20:17:35.785888 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.785862 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 20:17:35.820159 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.820130 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e76b596-bf27-4ec8-b864-8eb365f150fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b596-bf27-4ec8-b864-8eb365f150fc-plugins-config-volume\") pod \"6e76b596-bf27-4ec8-b864-8eb365f150fc\" (UID: \"6e76b596-bf27-4ec8-b864-8eb365f150fc\") " Nov 02 20:17:35.820348 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.820214 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e76b596-bf27-4ec8-b864-8eb365f150fc-kube-api-access-qsv8k\" (UniqueName: \"kubernetes.io/projected/6e76b596-bf27-4ec8-b864-8eb365f150fc-kube-api-access-qsv8k\") pod \"6e76b596-bf27-4ec8-b864-8eb365f150fc\" (UID: \"6e76b596-bf27-4ec8-b864-8eb365f150fc\") " Nov 02 20:17:35.820587 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.820562 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e76b596-bf27-4ec8-b864-8eb365f150fc-plugins-config-volume" pod "6e76b596-bf27-4ec8-b864-8eb365f150fc" (UID: "6e76b596-bf27-4ec8-b864-8eb365f150fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:35.820702 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.820683 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80659f4-faea-4edc-81e9-e525fed8afeb-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:35.820759 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.820709 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmpp\" (UniqueName: \"kubernetes.io/projected/e80659f4-faea-4edc-81e9-e525fed8afeb-kube-api-access-ghmpp\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:35.820759 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.820726 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e76b596-bf27-4ec8-b864-8eb365f150fc-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:35.822413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.822393 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e76b596-bf27-4ec8-b864-8eb365f150fc-kube-api-access-qsv8k" pod "6e76b596-bf27-4ec8-b864-8eb365f150fc" (UID: "6e76b596-bf27-4ec8-b864-8eb365f150fc"). InnerVolumeSpecName "kube-api-access-qsv8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:35.921918 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:35.921821 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsv8k\" (UniqueName: \"kubernetes.io/projected/6e76b596-bf27-4ec8-b864-8eb365f150fc-kube-api-access-qsv8k\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:36.037146 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:36.037108 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk"] Nov 02 20:17:36.040620 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:36.040590 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4848d2-6648f8f455-8bzsk"] Nov 02 20:17:36.746271 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:36.746184 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" Nov 02 20:17:36.746797 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:36.746184 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649" event={"ID":"6e76b596-bf27-4ec8-b864-8eb365f150fc","Type":"ContainerDied","Data":"83d0544ae49cf5d418eb790f50bca1982de6eed4cbe3a92e36ee19bfc41cd397"} Nov 02 20:17:36.746797 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:36.746341 5850 scope.go:122] "RemoveContainer" containerID="a8f064fe844b287a9bdfd040cb3060f6a2f90315cfab660310c0af90d1426c14" Nov 02 20:17:36.916995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:36.916955 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649"] Nov 02 20:17:36.979672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:36.979498 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4848d2-578b86d95-4l649"] Nov 02 20:17:37.814809 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:37.814777 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e76b596-bf27-4ec8-b864-8eb365f150fc" path="/var/lib/kubelet/pods/6e76b596-bf27-4ec8-b864-8eb365f150fc/volumes" Nov 02 20:17:37.815209 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:37.815171 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80659f4-faea-4edc-81e9-e525fed8afeb" path="/var/lib/kubelet/pods/e80659f4-faea-4edc-81e9-e525fed8afeb/volumes" Nov 02 20:17:38.392802 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.392762 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84"] Nov 02 20:17:38.393030 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.393009 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" podUID="5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" containerName="maas-api" containerID="cri-o://84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e" gracePeriod=30 Nov 02 20:17:38.510619 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.510583 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:38.651889 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.651822 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 20:17:38.753012 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.752968 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-kube-api-access-dp7d7\" (UniqueName: \"kubernetes.io/projected/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-kube-api-access-dp7d7\") pod \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " Nov 02 20:17:38.753227 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.753072 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls\") pod \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " Nov 02 20:17:38.753227 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.753128 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-openshift-service-ca\") pod \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " Nov 02 20:17:38.753356 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.753251 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-metrics-certs\") pod \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\" (UID: \"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3\") " Nov 02 20:17:38.753822 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.753790 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-openshift-service-ca" pod "5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" (UID: "5aa037a9-534d-46b8-bebb-a3f2a7fa11c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:38.757632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.757567 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-metrics-certs" pod "5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" (UID: "5aa037a9-534d-46b8-bebb-a3f2a7fa11c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:38.758030 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.757997 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls" pod "5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" (UID: "5aa037a9-534d-46b8-bebb-a3f2a7fa11c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:38.758741 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.758713 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-kube-api-access-dp7d7" pod "5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" (UID: "5aa037a9-534d-46b8-bebb-a3f2a7fa11c3"). InnerVolumeSpecName "kube-api-access-dp7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:38.772832 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.772789 5850 generic.go:356] "Generic (PLEG): container finished" podID="5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" containerID="84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e" exitCode=0 Nov 02 20:17:38.773005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.772839 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" event={"ID":"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3","Type":"ContainerDied","Data":"84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e"} Nov 02 20:17:38.773005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.772920 5850 scope.go:122] "RemoveContainer" containerID="84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e" Nov 02 20:17:38.773005 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.772854 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" Nov 02 20:17:38.773752 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.773682 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84" event={"ID":"5aa037a9-534d-46b8-bebb-a3f2a7fa11c3","Type":"ContainerDied","Data":"be13471ff3fb41a01149649d8ba57ba5ab30847b869c5ada97374abeb20d22d5"} Nov 02 20:17:38.788051 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.788028 5850 scope.go:122] "RemoveContainer" containerID="84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e" Nov 02 20:17:38.788435 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:38.788410 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e\": container with ID starting with 84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e not found: ID does not exist" containerID="84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e" Nov 02 20:17:38.788545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.788448 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e"} err="failed to get container status \"84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e\": rpc error: code = NotFound desc = could not find container \"84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e\": container with ID starting with 84d2c842ef1534c98ba0251fe471009d9d0707fdfa19b12d11c971a33e2a411e not found: ID does not exist" Nov 02 20:17:38.854807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.854615 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:38.854807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.854686 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp7d7\" (UniqueName: \"kubernetes.io/projected/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-kube-api-access-dp7d7\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:38.854807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.854712 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:38.854807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.854728 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:38.947141 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.946914 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84"] Nov 02 20:17:38.948527 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:38.948502 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-43ceaa-5f4b886cf4-6ld84"] Nov 02 20:17:39.818674 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:39.815777 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" path="/var/lib/kubelet/pods/5aa037a9-534d-46b8-bebb-a3f2a7fa11c3/volumes" Nov 02 20:17:39.925433 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:39.925401 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:40.130804 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:40.130762 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:40.303272 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:40.303242 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:40.796103 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:40.796069 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:44.164949 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:44.164911 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:44.760874 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:44.760837 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:17:45.254011 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:45.253974 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:45.558048 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:45.557960 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:45.816891 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:45.816813 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:46.394327 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:46.394291 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:46.720240 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:46.720159 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:46.903894 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:46.903856 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:47.061883 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:47.061779 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:47.252306 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:47.252275 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:47.686570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:47.686531 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:51.541649 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:51.541616 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:52.257153 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:52.257118 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:52.413228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:52.413195 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:52.567568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:52.567466 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6"] Nov 02 20:17:52.567979 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:52.567690 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" containerID="cri-o://2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262" gracePeriod=30 Nov 02 20:17:52.595647 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:52.595614 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:53.511350 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:53.511316 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:53.665949 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:53.665913 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:53.868261 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:53.868197 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:54.348447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:54.348361 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:56.178021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:56.177980 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:57.835453 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.835385 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 20:17:57.876700 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.876666 5850 generic.go:356] "Generic (PLEG): container finished" podID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerID="2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262" exitCode=0 Nov 02 20:17:57.876910 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.876725 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" Nov 02 20:17:57.876910 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.876747 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" event={"ID":"4e1564eb-9776-44b3-9a46-310ff4d64841","Type":"ContainerDied","Data":"2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262"} Nov 02 20:17:57.876910 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.876805 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6" event={"ID":"4e1564eb-9776-44b3-9a46-310ff4d64841","Type":"ContainerDied","Data":"11108a9bfd7a453866a69fbc8940627f9ba0a9e50096ad734236a6545c620498"} Nov 02 20:17:57.876910 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.876829 5850 scope.go:122] "RemoveContainer" containerID="2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262" Nov 02 20:17:57.885721 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.885698 5850 scope.go:122] "RemoveContainer" containerID="13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd" Nov 02 20:17:57.894984 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.894965 5850 scope.go:122] "RemoveContainer" containerID="2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262" Nov 02 20:17:57.895294 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:57.895273 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262\": container with ID starting with 2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262 not found: ID does not exist" containerID="2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262" Nov 02 20:17:57.895343 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.895305 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262"} err="failed to get container status \"2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262\": rpc error: code = NotFound desc = could not find container \"2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262\": container with ID starting with 2c7f47f3a0ae09ab5050cf7198e98438d020ffe159f657752cce020e85296262 not found: ID does not exist" Nov 02 20:17:57.895343 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.895326 5850 scope.go:122] "RemoveContainer" containerID="13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd" Nov 02 20:17:57.895624 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:17:57.895602 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd\": container with ID starting with 13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd not found: ID does not exist" containerID="13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd" Nov 02 20:17:57.895670 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.895631 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd"} err="failed to get container status \"13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd\": rpc error: code = NotFound desc = could not find container \"13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd\": container with ID starting with 13cca873a1ebdc9f69e6e77abbfbac26da57eea1140decc42c3740d1e6ada4bd not found: ID does not exist" Nov 02 20:17:57.959284 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959244 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-credential-socket\") pod \"4e1564eb-9776-44b3-9a46-310ff4d64841\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " Nov 02 20:17:57.959505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959299 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-envoy\") pod \"4e1564eb-9776-44b3-9a46-310ff4d64841\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " Nov 02 20:17:57.959505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959319 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-socket\") pod \"4e1564eb-9776-44b3-9a46-310ff4d64841\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " Nov 02 20:17:57.959505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959362 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-podinfo\") pod \"4e1564eb-9776-44b3-9a46-310ff4d64841\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " Nov 02 20:17:57.959505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959386 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-kube-api-access-r4kcr\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-kube-api-access-r4kcr\") pod \"4e1564eb-9776-44b3-9a46-310ff4d64841\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " Nov 02 20:17:57.959505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959415 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1564eb-9776-44b3-9a46-310ff4d64841-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1564eb-9776-44b3-9a46-310ff4d64841-istiod-ca-cert\") pod \"4e1564eb-9776-44b3-9a46-310ff4d64841\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " Nov 02 20:17:57.959800 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959540 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-certs\") pod \"4e1564eb-9776-44b3-9a46-310ff4d64841\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " Nov 02 20:17:57.959800 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959587 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-data\") pod \"4e1564eb-9776-44b3-9a46-310ff4d64841\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " Nov 02 20:17:57.959800 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959616 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-token\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-token\") pod \"4e1564eb-9776-44b3-9a46-310ff4d64841\" (UID: \"4e1564eb-9776-44b3-9a46-310ff4d64841\") " Nov 02 20:17:57.959800 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959681 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-credential-socket" pod "4e1564eb-9776-44b3-9a46-310ff4d64841" (UID: "4e1564eb-9776-44b3-9a46-310ff4d64841"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:57.959800 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959741 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-socket" pod "4e1564eb-9776-44b3-9a46-310ff4d64841" (UID: "4e1564eb-9776-44b3-9a46-310ff4d64841"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:57.960073 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959930 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-certs" pod "4e1564eb-9776-44b3-9a46-310ff4d64841" (UID: "4e1564eb-9776-44b3-9a46-310ff4d64841"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:57.960073 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959960 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1564eb-9776-44b3-9a46-310ff4d64841-istiod-ca-cert" pod "4e1564eb-9776-44b3-9a46-310ff4d64841" (UID: "4e1564eb-9776-44b3-9a46-310ff4d64841"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:57.960073 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.959984 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-data" pod "4e1564eb-9776-44b3-9a46-310ff4d64841" (UID: "4e1564eb-9776-44b3-9a46-310ff4d64841"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:57.960235 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.960095 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:57.960235 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.960114 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:57.960235 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.960129 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:57.960235 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.960143 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:57.960235 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.960159 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1564eb-9776-44b3-9a46-310ff4d64841-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:57.961874 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.961852 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-podinfo" pod "4e1564eb-9776-44b3-9a46-310ff4d64841" (UID: "4e1564eb-9776-44b3-9a46-310ff4d64841"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:17:57.961994 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.961874 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-token" pod "4e1564eb-9776-44b3-9a46-310ff4d64841" (UID: "4e1564eb-9776-44b3-9a46-310ff4d64841"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:57.962107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.962087 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-envoy" pod "4e1564eb-9776-44b3-9a46-310ff4d64841" (UID: "4e1564eb-9776-44b3-9a46-310ff4d64841"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:57.962173 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:57.962102 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-kube-api-access-r4kcr" pod "4e1564eb-9776-44b3-9a46-310ff4d64841" (UID: "4e1564eb-9776-44b3-9a46-310ff4d64841"). InnerVolumeSpecName "kube-api-access-r4kcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:58.060751 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:58.060718 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:58.060751 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:58.060747 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:58.060751 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:58.060756 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1564eb-9776-44b3-9a46-310ff4d64841-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:58.061017 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:58.060767 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kcr\" (UniqueName: \"kubernetes.io/projected/4e1564eb-9776-44b3-9a46-310ff4d64841-kube-api-access-r4kcr\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:58.301710 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:58.301672 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6"] Nov 02 20:17:58.311742 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:58.307533 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4848d2-openshift-default-859f6c588f-dsmb6"] Nov 02 20:17:59.301305 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:59.301259 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g"] Nov 02 20:17:59.394268 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:59.394219 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:59.814312 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:17:59.814274 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" path="/var/lib/kubelet/pods/4e1564eb-9776-44b3-9a46-310ff4d64841/volumes" Nov 02 20:18:00.737686 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:00.737653 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:04.875654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:04.875602 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7"] Nov 02 20:18:04.990486 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:04.990445 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq"] Nov 02 20:18:05.133542 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:05.133414 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:05.296036 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:05.296001 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:05.487753 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:05.487664 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:07.860821 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:07.860707 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:08.011085 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:08.011016 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:08.823656 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:08.823614 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7"] Nov 02 20:18:08.823879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:08.823843 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" podUID="680240bb-879b-46bf-b773-78cd4e768a26" containerName="maas-api" containerID="cri-o://941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79" gracePeriod=30 Nov 02 20:18:08.942581 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:08.942541 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:09.097868 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.097839 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 20:18:09.279449 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.279410 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680240bb-879b-46bf-b773-78cd4e768a26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680240bb-879b-46bf-b773-78cd4e768a26-openshift-service-ca\") pod \"680240bb-879b-46bf-b773-78cd4e768a26\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " Nov 02 20:18:09.279660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.279564 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680240bb-879b-46bf-b773-78cd4e768a26-kube-api-access-5bs4d\" (UniqueName: \"kubernetes.io/projected/680240bb-879b-46bf-b773-78cd4e768a26-kube-api-access-5bs4d\") pod \"680240bb-879b-46bf-b773-78cd4e768a26\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " Nov 02 20:18:09.279660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.279590 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls\") pod \"680240bb-879b-46bf-b773-78cd4e768a26\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " Nov 02 20:18:09.279660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.279623 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-metrics-certs\") pod \"680240bb-879b-46bf-b773-78cd4e768a26\" (UID: \"680240bb-879b-46bf-b773-78cd4e768a26\") " Nov 02 20:18:09.279922 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.279878 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680240bb-879b-46bf-b773-78cd4e768a26-openshift-service-ca" pod "680240bb-879b-46bf-b773-78cd4e768a26" (UID: "680240bb-879b-46bf-b773-78cd4e768a26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:09.281809 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.281774 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-metrics-certs" pod "680240bb-879b-46bf-b773-78cd4e768a26" (UID: "680240bb-879b-46bf-b773-78cd4e768a26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:09.281909 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.281831 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680240bb-879b-46bf-b773-78cd4e768a26-kube-api-access-5bs4d" pod "680240bb-879b-46bf-b773-78cd4e768a26" (UID: "680240bb-879b-46bf-b773-78cd4e768a26"). InnerVolumeSpecName "kube-api-access-5bs4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:09.281909 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.281842 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls" pod "680240bb-879b-46bf-b773-78cd4e768a26" (UID: "680240bb-879b-46bf-b773-78cd4e768a26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:09.381360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.381322 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680240bb-879b-46bf-b773-78cd4e768a26-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:09.381360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.381353 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bs4d\" (UniqueName: \"kubernetes.io/projected/680240bb-879b-46bf-b773-78cd4e768a26-kube-api-access-5bs4d\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:09.381360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.381363 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:09.381677 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.381372 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680240bb-879b-46bf-b773-78cd4e768a26-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:09.876262 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.876211 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" podUID="365d84dc-cf64-4d85-9d5b-05647ab37fd1" containerName="payload-processing" containerID="cri-o://fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9" gracePeriod=25 Nov 02 20:18:09.943941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.943905 5850 generic.go:356] "Generic (PLEG): container finished" podID="680240bb-879b-46bf-b773-78cd4e768a26" containerID="941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79" exitCode=0 Nov 02 20:18:09.944408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.943953 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" Nov 02 20:18:09.944408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.943982 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" event={"ID":"680240bb-879b-46bf-b773-78cd4e768a26","Type":"ContainerDied","Data":"941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79"} Nov 02 20:18:09.944408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.944050 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7" event={"ID":"680240bb-879b-46bf-b773-78cd4e768a26","Type":"ContainerDied","Data":"ff607c80c942805b6fe792a4e0b4b2f5ee871b1d614ed4d9e0f01b5d1dacfce2"} Nov 02 20:18:09.944408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.944081 5850 scope.go:122] "RemoveContainer" containerID="941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79" Nov 02 20:18:09.987510 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.987460 5850 scope.go:122] "RemoveContainer" containerID="941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79" Nov 02 20:18:09.987854 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:09.987829 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79\": container with ID starting with 941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79 not found: ID does not exist" containerID="941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79" Nov 02 20:18:09.987943 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.987862 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79"} err="failed to get container status \"941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79\": rpc error: code = NotFound desc = could not find container \"941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79\": container with ID starting with 941b2fac479ccf22bdddef9d034b9a44035322d685720219eb8a08055cc87a79 not found: ID does not exist" Nov 02 20:18:09.990989 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:09.990964 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" podUID="9ba308b4-8fec-4518-a5c3-245ed4f543f7" containerName="payload-pre-processing" containerID="cri-o://03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a" gracePeriod=25 Nov 02 20:18:10.114315 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.114284 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7"] Nov 02 20:18:10.116739 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.116714 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-282875-6dbbfc89bd-r8lp7"] Nov 02 20:18:10.131249 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.131220 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 20:18:10.189612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.189582 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365d84dc-cf64-4d85-9d5b-05647ab37fd1-kube-api-access-xcpjp\" (UniqueName: \"kubernetes.io/projected/365d84dc-cf64-4d85-9d5b-05647ab37fd1-kube-api-access-xcpjp\") pod \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\" (UID: \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\") " Nov 02 20:18:10.189796 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.189755 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365d84dc-cf64-4d85-9d5b-05647ab37fd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365d84dc-cf64-4d85-9d5b-05647ab37fd1-plugins-config-volume\") pod \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\" (UID: \"365d84dc-cf64-4d85-9d5b-05647ab37fd1\") " Nov 02 20:18:10.190319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.190296 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365d84dc-cf64-4d85-9d5b-05647ab37fd1-plugins-config-volume" pod "365d84dc-cf64-4d85-9d5b-05647ab37fd1" (UID: "365d84dc-cf64-4d85-9d5b-05647ab37fd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:10.191868 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.191843 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365d84dc-cf64-4d85-9d5b-05647ab37fd1-kube-api-access-xcpjp" pod "365d84dc-cf64-4d85-9d5b-05647ab37fd1" (UID: "365d84dc-cf64-4d85-9d5b-05647ab37fd1"). InnerVolumeSpecName "kube-api-access-xcpjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:10.241012 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.240980 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 20:18:10.291446 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.291414 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba308b4-8fec-4518-a5c3-245ed4f543f7-kube-api-access-7jds2\" (UniqueName: \"kubernetes.io/projected/9ba308b4-8fec-4518-a5c3-245ed4f543f7-kube-api-access-7jds2\") pod \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\" (UID: \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\") " Nov 02 20:18:10.291629 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.291516 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba308b4-8fec-4518-a5c3-245ed4f543f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba308b4-8fec-4518-a5c3-245ed4f543f7-plugins-config-volume\") pod \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\" (UID: \"9ba308b4-8fec-4518-a5c3-245ed4f543f7\") " Nov 02 20:18:10.291816 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.291804 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcpjp\" (UniqueName: \"kubernetes.io/projected/365d84dc-cf64-4d85-9d5b-05647ab37fd1-kube-api-access-xcpjp\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:10.291857 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.291819 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365d84dc-cf64-4d85-9d5b-05647ab37fd1-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:10.291892 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.291856 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba308b4-8fec-4518-a5c3-245ed4f543f7-plugins-config-volume" pod "9ba308b4-8fec-4518-a5c3-245ed4f543f7" (UID: "9ba308b4-8fec-4518-a5c3-245ed4f543f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:10.293586 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.293564 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba308b4-8fec-4518-a5c3-245ed4f543f7-kube-api-access-7jds2" pod "9ba308b4-8fec-4518-a5c3-245ed4f543f7" (UID: "9ba308b4-8fec-4518-a5c3-245ed4f543f7"). InnerVolumeSpecName "kube-api-access-7jds2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:10.370995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.370961 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:10.393380 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.393342 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba308b4-8fec-4518-a5c3-245ed4f543f7-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:10.393380 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.393378 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jds2\" (UniqueName: \"kubernetes.io/projected/9ba308b4-8fec-4518-a5c3-245ed4f543f7-kube-api-access-7jds2\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:10.618604 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.618567 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:10.778010 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.777930 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:10.950014 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.949973 5850 generic.go:356] "Generic (PLEG): container finished" podID="9ba308b4-8fec-4518-a5c3-245ed4f543f7" containerID="03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a" exitCode=0 Nov 02 20:18:10.950466 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.950053 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" Nov 02 20:18:10.950466 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.950062 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" event={"ID":"9ba308b4-8fec-4518-a5c3-245ed4f543f7","Type":"ContainerDied","Data":"03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a"} Nov 02 20:18:10.950466 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.950155 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq" event={"ID":"9ba308b4-8fec-4518-a5c3-245ed4f543f7","Type":"ContainerDied","Data":"29fe50273f1469a40689854cc8d75f48de485d05401b93dce2f22fc9fe39ce78"} Nov 02 20:18:10.950466 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.950184 5850 scope.go:122] "RemoveContainer" containerID="03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a" Nov 02 20:18:10.951469 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.951445 5850 generic.go:356] "Generic (PLEG): container finished" podID="365d84dc-cf64-4d85-9d5b-05647ab37fd1" containerID="fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9" exitCode=0 Nov 02 20:18:10.951571 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.951529 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" Nov 02 20:18:10.951571 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.951535 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" event={"ID":"365d84dc-cf64-4d85-9d5b-05647ab37fd1","Type":"ContainerDied","Data":"fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9"} Nov 02 20:18:10.951673 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.951588 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7" event={"ID":"365d84dc-cf64-4d85-9d5b-05647ab37fd1","Type":"ContainerDied","Data":"f7074b36ff7c8d1947557c7a8ec15ee935d66ac8fef2424934d1b92635208631"} Nov 02 20:18:10.959783 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.959765 5850 scope.go:122] "RemoveContainer" containerID="03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a" Nov 02 20:18:10.960121 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:10.960099 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a\": container with ID starting with 03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a not found: ID does not exist" containerID="03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a" Nov 02 20:18:10.960189 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.960135 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a"} err="failed to get container status \"03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a\": rpc error: code = NotFound desc = could not find container \"03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a\": container with ID starting with 03f49ccee8b75332797d28d76409b58af00c6f1af6baa7382340383769e1e67a not found: ID does not exist" Nov 02 20:18:10.960189 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.960161 5850 scope.go:122] "RemoveContainer" containerID="fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9" Nov 02 20:18:10.969048 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.969027 5850 scope.go:122] "RemoveContainer" containerID="fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9" Nov 02 20:18:10.969370 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:10.969345 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9\": container with ID starting with fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9 not found: ID does not exist" containerID="fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9" Nov 02 20:18:10.969496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:10.969377 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9"} err="failed to get container status \"fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9\": rpc error: code = NotFound desc = could not find container \"fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9\": container with ID starting with fdcd925639823fc77bef75e27c86c6e1b9bd27d180f23e3169212734b9196ef9 not found: ID does not exist" Nov 02 20:18:11.120756 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:11.120717 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq"] Nov 02 20:18:11.124121 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:11.124092 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-282875-59c5467f7f-jcntq"] Nov 02 20:18:11.237253 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:11.237216 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:11.240175 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:11.240148 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7"] Nov 02 20:18:11.243692 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:11.243668 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-282875-56ffd9d665-gb7s7"] Nov 02 20:18:11.815720 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:11.815685 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365d84dc-cf64-4d85-9d5b-05647ab37fd1" path="/var/lib/kubelet/pods/365d84dc-cf64-4d85-9d5b-05647ab37fd1/volumes" Nov 02 20:18:11.816127 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:11.816114 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680240bb-879b-46bf-b773-78cd4e768a26" path="/var/lib/kubelet/pods/680240bb-879b-46bf-b773-78cd4e768a26/volumes" Nov 02 20:18:11.816455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:11.816441 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba308b4-8fec-4518-a5c3-245ed4f543f7" path="/var/lib/kubelet/pods/9ba308b4-8fec-4518-a5c3-245ed4f543f7/volumes" Nov 02 20:18:14.321866 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.321822 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" containerID="cri-o://63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c" gracePeriod=45 Nov 02 20:18:14.572286 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.572212 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 20:18:14.646047 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646009 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-kube-api-access-xt77m\" (UniqueName: \"kubernetes.io/projected/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-kube-api-access-xt77m\") pod \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " Nov 02 20:18:14.646230 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646057 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tmp-dir\") pod \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " Nov 02 20:18:14.646230 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646082 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-model-cache\") pod \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " Nov 02 20:18:14.646230 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646154 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tls-certs\") pod \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " Nov 02 20:18:14.646230 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646213 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-home\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-home\") pod \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " Nov 02 20:18:14.646512 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646281 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-dshm\") pod \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\" (UID: \"cdc04263-4ed8-4e57-893e-d6ce55d7cd53\") " Nov 02 20:18:14.646512 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646361 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tmp-dir" pod "cdc04263-4ed8-4e57-893e-d6ce55d7cd53" (UID: "cdc04263-4ed8-4e57-893e-d6ce55d7cd53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.646512 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646376 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-model-cache" pod "cdc04263-4ed8-4e57-893e-d6ce55d7cd53" (UID: "cdc04263-4ed8-4e57-893e-d6ce55d7cd53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.646679 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646572 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-home" pod "cdc04263-4ed8-4e57-893e-d6ce55d7cd53" (UID: "cdc04263-4ed8-4e57-893e-d6ce55d7cd53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.646741 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646719 5850 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-home\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.646790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646740 5850 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tmp-dir\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.646790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.646755 5850 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-model-cache\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.648437 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.648410 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tls-certs" pod "cdc04263-4ed8-4e57-893e-d6ce55d7cd53" (UID: "cdc04263-4ed8-4e57-893e-d6ce55d7cd53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:14.648572 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.648440 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-dshm" pod "cdc04263-4ed8-4e57-893e-d6ce55d7cd53" (UID: "cdc04263-4ed8-4e57-893e-d6ce55d7cd53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.648572 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.648495 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-kube-api-access-xt77m" pod "cdc04263-4ed8-4e57-893e-d6ce55d7cd53" (UID: "cdc04263-4ed8-4e57-893e-d6ce55d7cd53"). InnerVolumeSpecName "kube-api-access-xt77m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:14.748544 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.748506 5850 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-dshm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.748745 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.748550 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt77m\" (UniqueName: \"kubernetes.io/projected/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-kube-api-access-xt77m\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.748745 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.748568 5850 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc04263-4ed8-4e57-893e-d6ce55d7cd53-tls-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.765347 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:14.765306 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:18:14.975334 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.975301 5850 generic.go:356] "Generic (PLEG): container finished" podID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerID="63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c" exitCode=0 Nov 02 20:18:14.975538 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.975348 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" Nov 02 20:18:14.975538 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.975383 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" event={"ID":"cdc04263-4ed8-4e57-893e-d6ce55d7cd53","Type":"ContainerDied","Data":"63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c"} Nov 02 20:18:14.975538 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.975415 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g" event={"ID":"cdc04263-4ed8-4e57-893e-d6ce55d7cd53","Type":"ContainerDied","Data":"91f4b77c74f7735a8af3831d82e28db6d7552ee2a51e4339f86799306a1748f4"} Nov 02 20:18:14.975538 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.975431 5850 scope.go:122] "RemoveContainer" containerID="63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c" Nov 02 20:18:14.985008 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.984978 5850 scope.go:122] "RemoveContainer" containerID="63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c" Nov 02 20:18:14.985298 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:14.985277 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c\": container with ID starting with 63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c not found: ID does not exist" containerID="63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c" Nov 02 20:18:14.985368 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:14.985312 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c"} err="failed to get container status \"63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c\": rpc error: code = NotFound desc = could not find container \"63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c\": container with ID starting with 63f1af27676f0088dfc5f21267576da091543bd468676dbe2cec6af4412d2e8c not found: ID does not exist" Nov 02 20:18:15.142297 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:15.142259 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g"] Nov 02 20:18:15.146049 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:15.146020 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/simulator-w6-e9c144-kserve-775856685f-wj75g"] Nov 02 20:18:15.252446 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:15.252360 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:15.820898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:15.820865 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" path="/var/lib/kubelet/pods/cdc04263-4ed8-4e57-893e-d6ce55d7cd53/volumes" Nov 02 20:18:17.320777 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:17.320740 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:17.481639 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:17.481605 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:17.613349 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:17.613243 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52"] Nov 02 20:18:17.613557 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:17.613526 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" containerID="cri-o://6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d" gracePeriod=30 Nov 02 20:18:17.644255 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:17.644222 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:18.072921 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:18.072883 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:19.306680 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.306645 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg"] Nov 02 20:18:19.307090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307046 5850 cpu_manager.go:414] "containerMap: removing container" podUID="a1aeb7c5-beb6-4ef8-8701-14f2175a553b" containerName="main" Nov 02 20:18:19.307090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307056 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1aeb7c5-beb6-4ef8-8701-14f2175a553b" containerName="main" Nov 02 20:18:19.307090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307072 5850 cpu_manager.go:414] "containerMap: removing container" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerName="cleanup" Nov 02 20:18:19.307090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307078 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerName="cleanup" Nov 02 20:18:19.307090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307087 5850 cpu_manager.go:414] "containerMap: removing container" podUID="5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" containerName="maas-api" Nov 02 20:18:19.307090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307092 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" containerName="maas-api" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307099 5850 cpu_manager.go:414] "containerMap: removing container" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307105 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307115 5850 cpu_manager.go:414] "containerMap: removing container" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307121 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307127 5850 cpu_manager.go:414] "containerMap: removing container" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307132 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307141 5850 cpu_manager.go:414] "containerMap: removing container" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307146 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307153 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307157 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307163 5850 cpu_manager.go:414] "containerMap: removing container" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307167 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307174 5850 cpu_manager.go:414] "containerMap: removing container" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307178 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307185 5850 cpu_manager.go:414] "containerMap: removing container" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307190 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307201 5850 cpu_manager.go:414] "containerMap: removing container" podUID="3061fab2-9827-407a-951f-0220331f0a21" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307205 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3061fab2-9827-407a-951f-0220331f0a21" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307213 5850 cpu_manager.go:414] "containerMap: removing container" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307218 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307224 5850 cpu_manager.go:414] "containerMap: removing container" podUID="9ba308b4-8fec-4518-a5c3-245ed4f543f7" containerName="payload-pre-processing" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307229 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba308b4-8fec-4518-a5c3-245ed4f543f7" containerName="payload-pre-processing" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307237 5850 cpu_manager.go:414] "containerMap: removing container" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307242 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307251 5850 cpu_manager.go:414] "containerMap: removing container" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307255 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307261 5850 cpu_manager.go:414] "containerMap: removing container" podUID="365d84dc-cf64-4d85-9d5b-05647ab37fd1" containerName="payload-processing" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307265 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365d84dc-cf64-4d85-9d5b-05647ab37fd1" containerName="payload-processing" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307272 5850 cpu_manager.go:414] "containerMap: removing container" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307276 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307282 5850 cpu_manager.go:414] "containerMap: removing container" podUID="3061fab2-9827-407a-951f-0220331f0a21" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307286 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3061fab2-9827-407a-951f-0220331f0a21" containerName="cleanup" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307296 5850 cpu_manager.go:414] "containerMap: removing container" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307300 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" Nov 02 20:18:19.307289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307307 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307312 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307318 5850 cpu_manager.go:414] "containerMap: removing container" podUID="6e76b596-bf27-4ec8-b864-8eb365f150fc" containerName="payload-pre-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307323 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e76b596-bf27-4ec8-b864-8eb365f150fc" containerName="payload-pre-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307328 5850 cpu_manager.go:414] "containerMap: removing container" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307332 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307339 5850 cpu_manager.go:414] "containerMap: removing container" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307343 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307349 5850 cpu_manager.go:414] "containerMap: removing container" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307353 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307363 5850 cpu_manager.go:414] "containerMap: removing container" podUID="519e3ebf-d09c-45b5-a773-42b1dad0c314" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307367 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519e3ebf-d09c-45b5-a773-42b1dad0c314" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307375 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e80659f4-faea-4edc-81e9-e525fed8afeb" containerName="payload-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307380 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80659f4-faea-4edc-81e9-e525fed8afeb" containerName="payload-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307386 5850 cpu_manager.go:414] "containerMap: removing container" podUID="ddfcd7fe-218c-4efd-ba81-45f03d929f4b" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307390 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfcd7fe-218c-4efd-ba81-45f03d929f4b" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307398 5850 cpu_manager.go:414] "containerMap: removing container" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307402 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307408 5850 cpu_manager.go:414] "containerMap: removing container" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307411 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307417 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307421 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307427 5850 cpu_manager.go:414] "containerMap: removing container" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307430 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307438 5850 cpu_manager.go:414] "containerMap: removing container" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307442 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307448 5850 cpu_manager.go:414] "containerMap: removing container" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307451 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307457 5850 cpu_manager.go:414] "containerMap: removing container" podUID="265c981a-ccad-4e88-aeec-16b5efbce6ad" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307461 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265c981a-ccad-4e88-aeec-16b5efbce6ad" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307485 5850 cpu_manager.go:414] "containerMap: removing container" podUID="9bbff140-4b0c-492c-a2ba-7163dd68b5c3" containerName="payload-pre-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307493 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbff140-4b0c-492c-a2ba-7163dd68b5c3" containerName="payload-pre-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307501 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307506 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307513 5850 cpu_manager.go:414] "containerMap: removing container" podUID="f8e19696-fe64-401f-874e-68e07b37b454" containerName="payload-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307518 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e19696-fe64-401f-874e-68e07b37b454" containerName="payload-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307525 5850 cpu_manager.go:414] "containerMap: removing container" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307530 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307536 5850 cpu_manager.go:414] "containerMap: removing container" podUID="3061fab2-9827-407a-951f-0220331f0a21" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307540 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3061fab2-9827-407a-951f-0220331f0a21" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307548 5850 cpu_manager.go:414] "containerMap: removing container" podUID="680240bb-879b-46bf-b773-78cd4e768a26" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307553 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680240bb-879b-46bf-b773-78cd4e768a26" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307613 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307621 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="3061fab2-9827-407a-951f-0220331f0a21" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307628 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="680240bb-879b-46bf-b773-78cd4e768a26" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307635 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1aeb7c5-beb6-4ef8-8701-14f2175a553b" containerName="main" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307641 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307646 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa037a9-534d-46b8-bebb-a3f2a7fa11c3" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307651 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307658 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307663 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307668 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc04263-4ed8-4e57-893e-d6ce55d7cd53" containerName="main" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307672 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307677 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307682 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7cf89b2-daf1-44e0-a445-49d67c92861c" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307687 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307692 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="3061fab2-9827-407a-951f-0220331f0a21" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307696 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307701 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba308b4-8fec-4518-a5c3-245ed4f543f7" containerName="payload-pre-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307707 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="a632b5f5-8af8-4ec1-9d4d-7970df12d254" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307713 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc565ff-fda0-4743-add0-891392bc36dc" containerName="main" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307719 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307724 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="365d84dc-cf64-4d85-9d5b-05647ab37fd1" containerName="payload-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307729 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="3061fab2-9827-407a-951f-0220331f0a21" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307734 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307739 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307744 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e76b596-bf27-4ec8-b864-8eb365f150fc" containerName="payload-pre-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307750 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307756 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="519e3ebf-d09c-45b5-a773-42b1dad0c314" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307762 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80659f4-faea-4edc-81e9-e525fed8afeb" containerName="payload-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307768 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfcd7fe-218c-4efd-ba81-45f03d929f4b" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307773 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="50407317-673e-4dfa-9ee2-b70a95b97e31" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307778 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cb4481-97cd-44e3-a14a-01dbcac2752c" containerName="istio-proxy" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307784 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1445634e-bc86-41d4-972c-a61e056dc60b" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307789 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307795 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="130949ec-3bf8-456b-be8e-3968339e51a0" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307800 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" containerName="cleanup" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307805 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="265c981a-ccad-4e88-aeec-16b5efbce6ad" containerName="maas-api" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307810 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbff140-4b0c-492c-a2ba-7163dd68b5c3" containerName="payload-pre-processing" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307816 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef32e82-ae3e-4e7d-b263-68e656cf1784" containerName="main" Nov 02 20:18:19.308383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.307821 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e19696-fe64-401f-874e-68e07b37b454" containerName="payload-processing" Nov 02 20:18:19.312363 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.312342 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.367136 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.367097 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg"] Nov 02 20:18:19.398910 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.398874 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64cd7324-vdhbg\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.398910 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.398916 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6ks7\" (UniqueName: \"kubernetes.io/projected/580a7a33-0967-4f28-90b6-f9f3d117446b-kube-api-access-m6ks7\") pod \"maas-api-revoke-sub-64cd7324-vdhbg\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.399129 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.399072 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64cd7324-vdhbg\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.416686 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.416649 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:18:19.416854 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.416748 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:18:19.500425 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.500382 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64cd7324-vdhbg\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.500665 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.500470 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64cd7324-vdhbg\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.500665 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.500516 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6ks7\" (UniqueName: \"kubernetes.io/projected/580a7a33-0967-4f28-90b6-f9f3d117446b-kube-api-access-m6ks7\") pod \"maas-api-revoke-sub-64cd7324-vdhbg\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.501155 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.501131 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64cd7324-vdhbg\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.502927 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.502903 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64cd7324-vdhbg\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.558906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.558821 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6ks7\" (UniqueName: \"kubernetes.io/projected/580a7a33-0967-4f28-90b6-f9f3d117446b-kube-api-access-m6ks7\") pod \"maas-api-revoke-sub-64cd7324-vdhbg\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.625684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.625644 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:19.846260 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.846170 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:19.992908 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:18:19.992869 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580a7a33_0967_4f28_90b6_f9f3d117446b.slice/crio-5d941b35e0342ef82ab4ca4fb603d5adaf098e5a56c21a7803ca3df67af6f478 WatchSource:0}: Error finding container 5d941b35e0342ef82ab4ca4fb603d5adaf098e5a56c21a7803ca3df67af6f478: Status 404 returned error can't find the container with id 5d941b35e0342ef82ab4ca4fb603d5adaf098e5a56c21a7803ca3df67af6f478 Nov 02 20:18:19.994918 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:19.994893 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg"] Nov 02 20:18:20.003292 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:20.003261 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" event={"ID":"580a7a33-0967-4f28-90b6-f9f3d117446b","Type":"ContainerStarted","Data":"5d941b35e0342ef82ab4ca4fb603d5adaf098e5a56c21a7803ca3df67af6f478"} Nov 02 20:18:20.130328 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:20.130289 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:20.285146 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:20.285110 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:20.468575 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:20.468459 5850 patch_prober.go:29] interesting pod/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:18:20.468575 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:20.468550 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:18:21.009696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:21.009661 5850 generic.go:356] "Generic (PLEG): container finished" podID="580a7a33-0967-4f28-90b6-f9f3d117446b" containerID="21e0ceef1a453c2940169f1b4ea603dd4657e6fc15f715848d23c67db65219a2" exitCode=0 Nov 02 20:18:21.009917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:21.009749 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" event={"ID":"580a7a33-0967-4f28-90b6-f9f3d117446b","Type":"ContainerDied","Data":"21e0ceef1a453c2940169f1b4ea603dd4657e6fc15f715848d23c67db65219a2"} Nov 02 20:18:22.152538 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.152510 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:22.229541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.229507 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-openshift-ca\") pod \"580a7a33-0967-4f28-90b6-f9f3d117446b\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " Nov 02 20:18:22.229711 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.229556 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580a7a33-0967-4f28-90b6-f9f3d117446b-kube-api-access-m6ks7\" (UniqueName: \"kubernetes.io/projected/580a7a33-0967-4f28-90b6-f9f3d117446b-kube-api-access-m6ks7\") pod \"580a7a33-0967-4f28-90b6-f9f3d117446b\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " Nov 02 20:18:22.229711 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.229590 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-xks-ca\") pod \"580a7a33-0967-4f28-90b6-f9f3d117446b\" (UID: \"580a7a33-0967-4f28-90b6-f9f3d117446b\") " Nov 02 20:18:22.229902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.229875 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-openshift-ca" pod "580a7a33-0967-4f28-90b6-f9f3d117446b" (UID: "580a7a33-0967-4f28-90b6-f9f3d117446b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:22.231703 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.231676 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-xks-ca" pod "580a7a33-0967-4f28-90b6-f9f3d117446b" (UID: "580a7a33-0967-4f28-90b6-f9f3d117446b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:22.231798 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.231734 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580a7a33-0967-4f28-90b6-f9f3d117446b-kube-api-access-m6ks7" pod "580a7a33-0967-4f28-90b6-f9f3d117446b" (UID: "580a7a33-0967-4f28-90b6-f9f3d117446b"). InnerVolumeSpecName "kube-api-access-m6ks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:22.330760 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.330665 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.330760 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.330702 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6ks7\" (UniqueName: \"kubernetes.io/projected/580a7a33-0967-4f28-90b6-f9f3d117446b-kube-api-access-m6ks7\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.330760 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.330717 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580a7a33-0967-4f28-90b6-f9f3d117446b-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.853546 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.853518 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 20:18:22.937672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.937567 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-token\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-token\") pod \"ed8dc527-83b5-48cd-8726-efe1912a67c9\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " Nov 02 20:18:22.937672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.937641 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-credential-socket\") pod \"ed8dc527-83b5-48cd-8726-efe1912a67c9\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " Nov 02 20:18:22.937672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.937679 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8dc527-83b5-48cd-8726-efe1912a67c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8dc527-83b5-48cd-8726-efe1912a67c9-istiod-ca-cert\") pod \"ed8dc527-83b5-48cd-8726-efe1912a67c9\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " Nov 02 20:18:22.938056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.937719 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-socket\") pod \"ed8dc527-83b5-48cd-8726-efe1912a67c9\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " Nov 02 20:18:22.938056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.937746 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-envoy\") pod \"ed8dc527-83b5-48cd-8726-efe1912a67c9\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " Nov 02 20:18:22.938056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.937775 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-data\") pod \"ed8dc527-83b5-48cd-8726-efe1912a67c9\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " Nov 02 20:18:22.938056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.937851 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-certs\") pod \"ed8dc527-83b5-48cd-8726-efe1912a67c9\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " Nov 02 20:18:22.938056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.937902 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-podinfo\") pod \"ed8dc527-83b5-48cd-8726-efe1912a67c9\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " Nov 02 20:18:22.938056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.937940 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-kube-api-access-w84rc\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-kube-api-access-w84rc\") pod \"ed8dc527-83b5-48cd-8726-efe1912a67c9\" (UID: \"ed8dc527-83b5-48cd-8726-efe1912a67c9\") " Nov 02 20:18:22.938056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.938045 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-credential-socket" pod "ed8dc527-83b5-48cd-8726-efe1912a67c9" (UID: "ed8dc527-83b5-48cd-8726-efe1912a67c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:22.938420 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.938185 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-socket" pod "ed8dc527-83b5-48cd-8726-efe1912a67c9" (UID: "ed8dc527-83b5-48cd-8726-efe1912a67c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:22.938420 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.938199 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-data" pod "ed8dc527-83b5-48cd-8726-efe1912a67c9" (UID: "ed8dc527-83b5-48cd-8726-efe1912a67c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:22.938420 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.938221 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8dc527-83b5-48cd-8726-efe1912a67c9-istiod-ca-cert" pod "ed8dc527-83b5-48cd-8726-efe1912a67c9" (UID: "ed8dc527-83b5-48cd-8726-efe1912a67c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:22.938420 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.938356 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8dc527-83b5-48cd-8726-efe1912a67c9-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.938420 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.938378 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.938420 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.938392 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.938420 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.938406 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.938849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.938654 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-certs" pod "ed8dc527-83b5-48cd-8726-efe1912a67c9" (UID: "ed8dc527-83b5-48cd-8726-efe1912a67c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:22.940113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.940089 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-envoy" pod "ed8dc527-83b5-48cd-8726-efe1912a67c9" (UID: "ed8dc527-83b5-48cd-8726-efe1912a67c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:22.940206 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.940091 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-kube-api-access-w84rc" pod "ed8dc527-83b5-48cd-8726-efe1912a67c9" (UID: "ed8dc527-83b5-48cd-8726-efe1912a67c9"). InnerVolumeSpecName "kube-api-access-w84rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:22.940306 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.940290 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-token" pod "ed8dc527-83b5-48cd-8726-efe1912a67c9" (UID: "ed8dc527-83b5-48cd-8726-efe1912a67c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:22.940390 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:22.940374 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-podinfo" pod "ed8dc527-83b5-48cd-8726-efe1912a67c9" (UID: "ed8dc527-83b5-48cd-8726-efe1912a67c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:23.020168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.020135 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" Nov 02 20:18:23.020339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.020183 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg" event={"ID":"580a7a33-0967-4f28-90b6-f9f3d117446b","Type":"ContainerDied","Data":"5d941b35e0342ef82ab4ca4fb603d5adaf098e5a56c21a7803ca3df67af6f478"} Nov 02 20:18:23.020339 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.020241 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d941b35e0342ef82ab4ca4fb603d5adaf098e5a56c21a7803ca3df67af6f478" Nov 02 20:18:23.021863 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.021834 5850 generic.go:356] "Generic (PLEG): container finished" podID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerID="6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d" exitCode=0 Nov 02 20:18:23.021976 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.021889 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" Nov 02 20:18:23.021976 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.021909 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" event={"ID":"ed8dc527-83b5-48cd-8726-efe1912a67c9","Type":"ContainerDied","Data":"6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d"} Nov 02 20:18:23.021976 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.021960 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52" event={"ID":"ed8dc527-83b5-48cd-8726-efe1912a67c9","Type":"ContainerDied","Data":"dac5120c5569de0dea4124276b27016a0f0af0d6d455e47c39e3efd8be0fc952"} Nov 02 20:18:23.022147 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.022016 5850 scope.go:122] "RemoveContainer" containerID="6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d" Nov 02 20:18:23.036729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.036704 5850 scope.go:122] "RemoveContainer" containerID="5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b" Nov 02 20:18:23.039244 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.039221 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:23.039352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.039247 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:23.039352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.039263 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8dc527-83b5-48cd-8726-efe1912a67c9-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:23.039352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.039276 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8dc527-83b5-48cd-8726-efe1912a67c9-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:23.039352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.039289 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w84rc\" (UniqueName: \"kubernetes.io/projected/ed8dc527-83b5-48cd-8726-efe1912a67c9-kube-api-access-w84rc\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:23.045672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.045589 5850 scope.go:122] "RemoveContainer" containerID="6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d" Nov 02 20:18:23.045964 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:23.045940 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d\": container with ID starting with 6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d not found: ID does not exist" containerID="6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d" Nov 02 20:18:23.046001 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.045975 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d"} err="failed to get container status \"6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d\": rpc error: code = NotFound desc = could not find container \"6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d\": container with ID starting with 6b02e20a4eb5328e14255258a31ffdbcaf19599f5a20393c4e8743caa642950d not found: ID does not exist" Nov 02 20:18:23.046001 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.045993 5850 scope.go:122] "RemoveContainer" containerID="5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b" Nov 02 20:18:23.046249 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:23.046233 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b\": container with ID starting with 5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b not found: ID does not exist" containerID="5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b" Nov 02 20:18:23.046286 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.046255 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b"} err="failed to get container status \"5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b\": rpc error: code = NotFound desc = could not find container \"5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b\": container with ID starting with 5c323fea5dad87ab47a3be0725d745f3b93204970983309e38a97cd002f5d35b not found: ID does not exist" Nov 02 20:18:23.210697 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.210589 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:23.249783 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.249730 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52"] Nov 02 20:18:23.253014 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.252985 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-282875-openshift-default-768d7c48c9-5vl52"] Nov 02 20:18:23.813832 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:23.813797 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" path="/var/lib/kubelet/pods/ed8dc527-83b5-48cd-8726-efe1912a67c9/volumes" Nov 02 20:18:25.097712 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:25.097678 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:29.332284 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:29.332248 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:30.500752 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:30.500707 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:30.659068 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:30.659029 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:33.638296 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:33.638263 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:35.555397 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:35.555357 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:35.713387 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:35.713349 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:36.886172 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:36.886137 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:37.525341 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525304 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg"] Nov 02 20:18:37.525724 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525710 5850 cpu_manager.go:414] "containerMap: removing container" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" Nov 02 20:18:37.525776 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525725 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" Nov 02 20:18:37.525776 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525747 5850 cpu_manager.go:414] "containerMap: removing container" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" Nov 02 20:18:37.525776 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525753 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1564eb-9776-44b3-9a46-310ff4d64841" containerName="istio-proxy" Nov 02 20:18:37.525776 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525759 5850 cpu_manager.go:414] "containerMap: removing container" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" Nov 02 20:18:37.525776 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525765 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" Nov 02 20:18:37.525919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525790 5850 cpu_manager.go:414] "containerMap: removing container" podUID="580a7a33-0967-4f28-90b6-f9f3d117446b" containerName="revoke-keys" Nov 02 20:18:37.525919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525797 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580a7a33-0967-4f28-90b6-f9f3d117446b" containerName="revoke-keys" Nov 02 20:18:37.525919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525857 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="580a7a33-0967-4f28-90b6-f9f3d117446b" containerName="revoke-keys" Nov 02 20:18:37.525919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525865 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ebd778-efab-4d94-b73a-d00738bd2659" containerName="istio-proxy" Nov 02 20:18:37.525919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525872 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" Nov 02 20:18:37.525919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525880 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="007830fe-7b7f-4f0e-aae1-171927294d93" containerName="cleanup" Nov 02 20:18:37.525919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525885 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8dc527-83b5-48cd-8726-efe1912a67c9" containerName="istio-proxy" Nov 02 20:18:37.525919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.525891 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="844ad708-6b20-40a7-84b4-3b22ee9612c1" containerName="cleanup" Nov 02 20:18:37.528855 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.528837 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.578280 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.578247 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm"] Nov 02 20:18:37.578481 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.578453 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" podUID="82ae3868-9c4c-4c85-81ba-7633c542fa04" containerName="maas-api" containerID="cri-o://94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f" gracePeriod=30 Nov 02 20:18:37.588442 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.588404 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8062eb3d-sqnqg\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.588638 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.588452 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khbwb\" (UniqueName: \"kubernetes.io/projected/341add44-2864-4b3c-bdf2-7bb1712569c2-kube-api-access-khbwb\") pod \"maas-api-revoke-sub-8062eb3d-sqnqg\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.588638 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.588574 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8062eb3d-sqnqg\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.628527 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.628454 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:18:37.630788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.630750 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg"] Nov 02 20:18:37.636514 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.636457 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:18:37.689595 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.689561 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8062eb3d-sqnqg\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.689595 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.689597 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khbwb\" (UniqueName: \"kubernetes.io/projected/341add44-2864-4b3c-bdf2-7bb1712569c2-kube-api-access-khbwb\") pod \"maas-api-revoke-sub-8062eb3d-sqnqg\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.689786 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.689652 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8062eb3d-sqnqg\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.690383 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.690248 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8062eb3d-sqnqg\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.692456 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.692435 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8062eb3d-sqnqg\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.705282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.705225 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:37.748351 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.748315 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khbwb\" (UniqueName: \"kubernetes.io/projected/341add44-2864-4b3c-bdf2-7bb1712569c2-kube-api-access-khbwb\") pod \"maas-api-revoke-sub-8062eb3d-sqnqg\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.833396 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.833367 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 20:18:37.843282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.843250 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:37.891112 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.891076 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-metrics-certs\") pod \"82ae3868-9c4c-4c85-81ba-7633c542fa04\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " Nov 02 20:18:37.891725 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.891192 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls\") pod \"82ae3868-9c4c-4c85-81ba-7633c542fa04\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " Nov 02 20:18:37.891725 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.891283 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ae3868-9c4c-4c85-81ba-7633c542fa04-kube-api-access-hjghb\" (UniqueName: \"kubernetes.io/projected/82ae3868-9c4c-4c85-81ba-7633c542fa04-kube-api-access-hjghb\") pod \"82ae3868-9c4c-4c85-81ba-7633c542fa04\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " Nov 02 20:18:37.891725 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.891314 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ae3868-9c4c-4c85-81ba-7633c542fa04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ae3868-9c4c-4c85-81ba-7633c542fa04-openshift-service-ca\") pod \"82ae3868-9c4c-4c85-81ba-7633c542fa04\" (UID: \"82ae3868-9c4c-4c85-81ba-7633c542fa04\") " Nov 02 20:18:37.892304 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.892278 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ae3868-9c4c-4c85-81ba-7633c542fa04-openshift-service-ca" pod "82ae3868-9c4c-4c85-81ba-7633c542fa04" (UID: "82ae3868-9c4c-4c85-81ba-7633c542fa04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:37.893920 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.893877 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-metrics-certs" pod "82ae3868-9c4c-4c85-81ba-7633c542fa04" (UID: "82ae3868-9c4c-4c85-81ba-7633c542fa04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:37.894672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.894642 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls" pod "82ae3868-9c4c-4c85-81ba-7633c542fa04" (UID: "82ae3868-9c4c-4c85-81ba-7633c542fa04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:37.895564 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.895537 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ae3868-9c4c-4c85-81ba-7633c542fa04-kube-api-access-hjghb" pod "82ae3868-9c4c-4c85-81ba-7633c542fa04" (UID: "82ae3868-9c4c-4c85-81ba-7633c542fa04"). InnerVolumeSpecName "kube-api-access-hjghb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:37.992077 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.992036 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:37.992077 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.992073 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjghb\" (UniqueName: \"kubernetes.io/projected/82ae3868-9c4c-4c85-81ba-7633c542fa04-kube-api-access-hjghb\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:37.992077 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.992084 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ae3868-9c4c-4c85-81ba-7633c542fa04-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:37.992344 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:37.992092 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ae3868-9c4c-4c85-81ba-7633c542fa04-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.094367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.094277 5850 generic.go:356] "Generic (PLEG): container finished" podID="82ae3868-9c4c-4c85-81ba-7633c542fa04" containerID="94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f" exitCode=0 Nov 02 20:18:38.094367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.094324 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" Nov 02 20:18:38.094367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.094361 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" event={"ID":"82ae3868-9c4c-4c85-81ba-7633c542fa04","Type":"ContainerDied","Data":"94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f"} Nov 02 20:18:38.094684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.094417 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm" event={"ID":"82ae3868-9c4c-4c85-81ba-7633c542fa04","Type":"ContainerDied","Data":"5e63c0b6f5ec0e988c4477ffdfa03a08773d5518291c6923401c752815c3f16e"} Nov 02 20:18:38.094684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.094441 5850 scope.go:122] "RemoveContainer" containerID="94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f" Nov 02 20:18:38.104613 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.104590 5850 scope.go:122] "RemoveContainer" containerID="94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f" Nov 02 20:18:38.104921 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:38.104901 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f\": container with ID starting with 94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f not found: ID does not exist" containerID="94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f" Nov 02 20:18:38.104997 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.104937 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f"} err="failed to get container status \"94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f\": rpc error: code = NotFound desc = could not find container \"94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f\": container with ID starting with 94cc5a59bc0eb94321df72bf51ee873fdbbdddd845bbf4b783b8fa4bda163e2f not found: ID does not exist" Nov 02 20:18:38.213568 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:18:38.213532 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341add44_2864_4b3c_bdf2_7bb1712569c2.slice/crio-db52edc25d097ef62e9c4e5500bb24d9d64222202b97e47f61b09d01e525abcb WatchSource:0}: Error finding container db52edc25d097ef62e9c4e5500bb24d9d64222202b97e47f61b09d01e525abcb: Status 404 returned error can't find the container with id db52edc25d097ef62e9c4e5500bb24d9d64222202b97e47f61b09d01e525abcb Nov 02 20:18:38.214399 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.214377 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg"] Nov 02 20:18:38.264533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.264469 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm"] Nov 02 20:18:38.267915 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:38.267889 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e9c144-7d6b9f475c-zlqxm"] Nov 02 20:18:39.103161 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:39.103128 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" event={"ID":"341add44-2864-4b3c-bdf2-7bb1712569c2","Type":"ContainerStarted","Data":"069d5a7db2485f05e26a2fe39e12e18b9cfb68cc3bf88971519b2cf48e85ff84"} Nov 02 20:18:39.103672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:39.103177 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" event={"ID":"341add44-2864-4b3c-bdf2-7bb1712569c2","Type":"ContainerStarted","Data":"db52edc25d097ef62e9c4e5500bb24d9d64222202b97e47f61b09d01e525abcb"} Nov 02 20:18:39.184823 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:39.184792 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:39.223967 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:39.223912 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" podStartSLOduration=2.223896432 podStartE2EDuration="2.223896432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:39.223310635 +0000 UTC m=+2977.565699715" watchObservedRunningTime="2026-11-02 20:18:39.223896432 +0000 UTC m=+2977.566285505" Nov 02 20:18:39.404837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:39.404804 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:39.814186 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:39.814109 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ae3868-9c4c-4c85-81ba-7633c542fa04" path="/var/lib/kubelet/pods/82ae3868-9c4c-4c85-81ba-7633c542fa04/volumes" Nov 02 20:18:40.016427 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:40.016377 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:40.109421 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:40.109331 5850 generic.go:356] "Generic (PLEG): container finished" podID="341add44-2864-4b3c-bdf2-7bb1712569c2" containerID="069d5a7db2485f05e26a2fe39e12e18b9cfb68cc3bf88971519b2cf48e85ff84" exitCode=0 Nov 02 20:18:40.109822 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:40.109416 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" event={"ID":"341add44-2864-4b3c-bdf2-7bb1712569c2","Type":"ContainerDied","Data":"069d5a7db2485f05e26a2fe39e12e18b9cfb68cc3bf88971519b2cf48e85ff84"} Nov 02 20:18:40.292369 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:40.292309 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:40.492373 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:40.492340 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:41.130897 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.130859 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:41.252142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.252112 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:41.326037 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.326000 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-xks-ca\") pod \"341add44-2864-4b3c-bdf2-7bb1712569c2\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " Nov 02 20:18:41.326217 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.326158 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-openshift-ca\") pod \"341add44-2864-4b3c-bdf2-7bb1712569c2\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " Nov 02 20:18:41.326217 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.326177 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341add44-2864-4b3c-bdf2-7bb1712569c2-kube-api-access-khbwb\" (UniqueName: \"kubernetes.io/projected/341add44-2864-4b3c-bdf2-7bb1712569c2-kube-api-access-khbwb\") pod \"341add44-2864-4b3c-bdf2-7bb1712569c2\" (UID: \"341add44-2864-4b3c-bdf2-7bb1712569c2\") " Nov 02 20:18:41.326500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.326458 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-openshift-ca" pod "341add44-2864-4b3c-bdf2-7bb1712569c2" (UID: "341add44-2864-4b3c-bdf2-7bb1712569c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:41.328311 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.328287 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341add44-2864-4b3c-bdf2-7bb1712569c2-kube-api-access-khbwb" pod "341add44-2864-4b3c-bdf2-7bb1712569c2" (UID: "341add44-2864-4b3c-bdf2-7bb1712569c2"). InnerVolumeSpecName "kube-api-access-khbwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:41.328382 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.328363 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-xks-ca" pod "341add44-2864-4b3c-bdf2-7bb1712569c2" (UID: "341add44-2864-4b3c-bdf2-7bb1712569c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:41.427990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.427945 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:41.427990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.427987 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341add44-2864-4b3c-bdf2-7bb1712569c2-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:41.427990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:41.428000 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khbwb\" (UniqueName: \"kubernetes.io/projected/341add44-2864-4b3c-bdf2-7bb1712569c2-kube-api-access-khbwb\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:42.121409 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:42.121379 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" Nov 02 20:18:42.121409 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:42.121391 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg" event={"ID":"341add44-2864-4b3c-bdf2-7bb1712569c2","Type":"ContainerDied","Data":"db52edc25d097ef62e9c4e5500bb24d9d64222202b97e47f61b09d01e525abcb"} Nov 02 20:18:42.121675 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:42.121468 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db52edc25d097ef62e9c4e5500bb24d9d64222202b97e47f61b09d01e525abcb" Nov 02 20:18:42.670831 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:42.670795 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:42.827095 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:42.827062 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:44.761416 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:44.761374 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:18:46.163244 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:46.163204 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:46.341289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:46.341254 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:46.415350 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:46.415263 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v"] Nov 02 20:18:46.415559 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:46.415537 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" containerID="cri-o://43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72" gracePeriod=30 Nov 02 20:18:46.508717 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:46.508686 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:46.672537 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:46.672440 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:47.150089 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:47.150055 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:51.665174 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.665143 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 20:18:51.733354 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733270 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-certs\") pod \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " Nov 02 20:18:51.733354 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733319 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-credential-socket\") pod \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " Nov 02 20:18:51.733582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733369 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-envoy\") pod \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " Nov 02 20:18:51.733582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733418 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-data\") pod \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " Nov 02 20:18:51.733582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733451 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-podinfo\") pod \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " Nov 02 20:18:51.733582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733483 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istiod-ca-cert\") pod \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " Nov 02 20:18:51.733582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733555 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-token\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-token\") pod \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " Nov 02 20:18:51.733582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733582 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-kube-api-access-nlnlx\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-kube-api-access-nlnlx\") pod \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " Nov 02 20:18:51.733933 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733623 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-socket\") pod \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\" (UID: \"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a\") " Nov 02 20:18:51.733995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733615 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-certs" pod "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" (UID: "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:51.733995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733950 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-socket" pod "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" (UID: "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:51.733995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733974 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istiod-ca-cert" pod "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" (UID: "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:51.733995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733721 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-data" pod "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" (UID: "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:51.734166 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.733741 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-credential-socket" pod "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" (UID: "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:51.735804 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.735774 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-podinfo" pod "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" (UID: "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:51.735918 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.735844 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-envoy" pod "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" (UID: "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:51.735918 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.735908 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-kube-api-access-nlnlx" pod "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" (UID: "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a"). InnerVolumeSpecName "kube-api-access-nlnlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:51.736042 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.736028 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-token" pod "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" (UID: "fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:51.834774 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.834739 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:51.834774 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.834771 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:51.834774 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.834782 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:51.835018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.834791 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:51.835018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.834799 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:51.835018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.834810 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:51.835018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.834821 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlnlx\" (UniqueName: \"kubernetes.io/projected/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-kube-api-access-nlnlx\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:51.835018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.834830 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:51.835018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:51.834839 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:52.171854 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:52.171813 5850 generic.go:356] "Generic (PLEG): container finished" podID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerID="43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72" exitCode=0 Nov 02 20:18:52.172052 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:52.171866 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" Nov 02 20:18:52.172052 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:52.171887 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" event={"ID":"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a","Type":"ContainerDied","Data":"43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72"} Nov 02 20:18:52.172052 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:52.171944 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v" event={"ID":"fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a","Type":"ContainerDied","Data":"7341ee4adde783eb212b8ffd67490f9198fdcb6b1c6164af7a93bb82ab9f39f8"} Nov 02 20:18:52.172052 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:52.171970 5850 scope.go:122] "RemoveContainer" containerID="43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72" Nov 02 20:18:52.181097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:52.181075 5850 scope.go:122] "RemoveContainer" containerID="43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72" Nov 02 20:18:52.181383 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:18:52.181362 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72\": container with ID starting with 43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72 not found: ID does not exist" containerID="43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72" Nov 02 20:18:52.181448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:52.181397 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72"} err="failed to get container status \"43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72\": rpc error: code = NotFound desc = could not find container \"43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72\": container with ID starting with 43d30c5912ec2ac5a2b46d89f66ea316c9d52e49b9f2043f6a4e0350f89e9e72 not found: ID does not exist" Nov 02 20:18:52.354829 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:52.354787 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v"] Nov 02 20:18:52.368581 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:52.368542 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-spm9v"] Nov 02 20:18:53.648726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:53.648692 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:53.813760 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:53.813726 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" path="/var/lib/kubelet/pods/fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a/volumes" Nov 02 20:18:55.021726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:55.021684 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:55.279670 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:55.279578 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:55.903433 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:18:55.903405 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:01.688161 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:01.688118 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:01.855895 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:01.855862 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:02.277759 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:02.277727 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:02.437318 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:02.437284 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:02.539387 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:02.539306 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x"] Nov 02 20:19:02.539553 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:02.539525 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" containerID="cri-o://c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110" gracePeriod=30 Nov 02 20:19:02.590148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:02.590108 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:07.788513 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.788467 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 20:19:07.896621 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896581 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-certs\") pod \"9fc4e519-23d7-4081-89d1-4f69cc466f26\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " Nov 02 20:19:07.896835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896633 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-socket\") pod \"9fc4e519-23d7-4081-89d1-4f69cc466f26\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " Nov 02 20:19:07.896835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896670 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-podinfo\") pod \"9fc4e519-23d7-4081-89d1-4f69cc466f26\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " Nov 02 20:19:07.896835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896704 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-envoy\") pod \"9fc4e519-23d7-4081-89d1-4f69cc466f26\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " Nov 02 20:19:07.896835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896746 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-credential-socket\") pod \"9fc4e519-23d7-4081-89d1-4f69cc466f26\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " Nov 02 20:19:07.896835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896783 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-data\") pod \"9fc4e519-23d7-4081-89d1-4f69cc466f26\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " Nov 02 20:19:07.897105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896865 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-kube-api-access-nnxnv\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-kube-api-access-nnxnv\") pod \"9fc4e519-23d7-4081-89d1-4f69cc466f26\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " Nov 02 20:19:07.897105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896904 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc4e519-23d7-4081-89d1-4f69cc466f26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc4e519-23d7-4081-89d1-4f69cc466f26-istiod-ca-cert\") pod \"9fc4e519-23d7-4081-89d1-4f69cc466f26\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " Nov 02 20:19:07.897105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896905 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-certs" pod "9fc4e519-23d7-4081-89d1-4f69cc466f26" (UID: "9fc4e519-23d7-4081-89d1-4f69cc466f26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:07.897105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.896951 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-token\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-token\") pod \"9fc4e519-23d7-4081-89d1-4f69cc466f26\" (UID: \"9fc4e519-23d7-4081-89d1-4f69cc466f26\") " Nov 02 20:19:07.897105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.897023 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-socket" pod "9fc4e519-23d7-4081-89d1-4f69cc466f26" (UID: "9fc4e519-23d7-4081-89d1-4f69cc466f26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:07.897393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.897150 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-data" pod "9fc4e519-23d7-4081-89d1-4f69cc466f26" (UID: "9fc4e519-23d7-4081-89d1-4f69cc466f26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:07.897393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.897362 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:07.897393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.897387 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:07.897609 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.897412 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:07.897609 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.897524 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc4e519-23d7-4081-89d1-4f69cc466f26-istiod-ca-cert" pod "9fc4e519-23d7-4081-89d1-4f69cc466f26" (UID: "9fc4e519-23d7-4081-89d1-4f69cc466f26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:07.898282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.897100 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-credential-socket" pod "9fc4e519-23d7-4081-89d1-4f69cc466f26" (UID: "9fc4e519-23d7-4081-89d1-4f69cc466f26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:07.899356 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.899329 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-token" pod "9fc4e519-23d7-4081-89d1-4f69cc466f26" (UID: "9fc4e519-23d7-4081-89d1-4f69cc466f26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:07.899459 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.899335 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-envoy" pod "9fc4e519-23d7-4081-89d1-4f69cc466f26" (UID: "9fc4e519-23d7-4081-89d1-4f69cc466f26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:07.899459 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.899423 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-podinfo" pod "9fc4e519-23d7-4081-89d1-4f69cc466f26" (UID: "9fc4e519-23d7-4081-89d1-4f69cc466f26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:07.899669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.899646 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-kube-api-access-nnxnv" pod "9fc4e519-23d7-4081-89d1-4f69cc466f26" (UID: "9fc4e519-23d7-4081-89d1-4f69cc466f26"). InnerVolumeSpecName "kube-api-access-nnxnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:07.998664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.998621 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:07.998664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.998658 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxnv\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-kube-api-access-nnxnv\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:07.998664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.998669 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc4e519-23d7-4081-89d1-4f69cc466f26-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:07.998664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.998678 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:07.998940 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.998688 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:07.998940 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:07.998698 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc4e519-23d7-4081-89d1-4f69cc466f26-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:08.249380 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:08.249294 5850 generic.go:356] "Generic (PLEG): container finished" podID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerID="c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110" exitCode=0 Nov 02 20:19:08.249380 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:08.249342 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" Nov 02 20:19:08.249585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:08.249375 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" event={"ID":"9fc4e519-23d7-4081-89d1-4f69cc466f26","Type":"ContainerDied","Data":"c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110"} Nov 02 20:19:08.249585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:08.249435 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x" event={"ID":"9fc4e519-23d7-4081-89d1-4f69cc466f26","Type":"ContainerDied","Data":"f5a1c7d179c050cacda0710dcfccd3671d2927bc70ca8dec1c4ca4285c143fff"} Nov 02 20:19:08.249585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:08.249464 5850 scope.go:122] "RemoveContainer" containerID="c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110" Nov 02 20:19:08.259043 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:08.259021 5850 scope.go:122] "RemoveContainer" containerID="c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110" Nov 02 20:19:08.259334 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:19:08.259313 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110\": container with ID starting with c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110 not found: ID does not exist" containerID="c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110" Nov 02 20:19:08.259408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:08.259341 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110"} err="failed to get container status \"c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110\": rpc error: code = NotFound desc = could not find container \"c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110\": container with ID starting with c25e903c632a6d7e01f410f3f9a93d0660097b3f115318ced6ffd81a05df7110 not found: ID does not exist" Nov 02 20:19:08.418306 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:08.418263 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x"] Nov 02 20:19:08.469712 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:08.469674 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-5895x"] Nov 02 20:19:09.813129 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:09.813095 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" path="/var/lib/kubelet/pods/9fc4e519-23d7-4081-89d1-4f69cc466f26/volumes" Nov 02 20:19:14.759929 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:19:14.759890 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:19:20.242154 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:20.242119 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:20.575789 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:20.575696 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:31.162194 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162155 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f"] Nov 02 20:19:31.162664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162553 5850 cpu_manager.go:414] "containerMap: removing container" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" Nov 02 20:19:31.162664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162566 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" Nov 02 20:19:31.162664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162578 5850 cpu_manager.go:414] "containerMap: removing container" podUID="82ae3868-9c4c-4c85-81ba-7633c542fa04" containerName="maas-api" Nov 02 20:19:31.162664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162583 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ae3868-9c4c-4c85-81ba-7633c542fa04" containerName="maas-api" Nov 02 20:19:31.162664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162592 5850 cpu_manager.go:414] "containerMap: removing container" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" Nov 02 20:19:31.162664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162598 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" Nov 02 20:19:31.162664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162618 5850 cpu_manager.go:414] "containerMap: removing container" podUID="341add44-2864-4b3c-bdf2-7bb1712569c2" containerName="revoke-keys" Nov 02 20:19:31.162664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162623 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341add44-2864-4b3c-bdf2-7bb1712569c2" containerName="revoke-keys" Nov 02 20:19:31.162925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162684 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="341add44-2864-4b3c-bdf2-7bb1712569c2" containerName="revoke-keys" Nov 02 20:19:31.162925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162696 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3f4c78-7fcb-4ae0-91fb-a548e58c2c4a" containerName="istio-proxy" Nov 02 20:19:31.162925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162703 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ae3868-9c4c-4c85-81ba-7633c542fa04" containerName="maas-api" Nov 02 20:19:31.162925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.162708 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc4e519-23d7-4081-89d1-4f69cc466f26" containerName="istio-proxy" Nov 02 20:19:31.165701 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.165684 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.214612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.214566 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178183e7-e489-4178-99c3-5309122770e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bjr5f\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.214814 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.214709 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178183e7-e489-4178-99c3-5309122770e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bjr5f\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.214814 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.214745 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87qkz\" (UniqueName: \"kubernetes.io/projected/178183e7-e489-4178-99c3-5309122770e1-kube-api-access-87qkz\") pod \"maas-api-revoke-sub-85237e45-bjr5f\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.225177 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.225133 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f"] Nov 02 20:19:31.274824 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.274785 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:19:31.274824 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.274821 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:19:31.316178 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.316127 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178183e7-e489-4178-99c3-5309122770e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bjr5f\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.316178 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.316173 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87qkz\" (UniqueName: \"kubernetes.io/projected/178183e7-e489-4178-99c3-5309122770e1-kube-api-access-87qkz\") pod \"maas-api-revoke-sub-85237e45-bjr5f\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.316404 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.316240 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178183e7-e489-4178-99c3-5309122770e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bjr5f\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.316936 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.316916 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178183e7-e489-4178-99c3-5309122770e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bjr5f\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.318545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.318521 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178183e7-e489-4178-99c3-5309122770e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bjr5f\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.374508 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.374459 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87qkz\" (UniqueName: \"kubernetes.io/projected/178183e7-e489-4178-99c3-5309122770e1-kube-api-access-87qkz\") pod \"maas-api-revoke-sub-85237e45-bjr5f\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.478603 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.478511 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:31.631090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.631052 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts"] Nov 02 20:19:31.635753 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.635728 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.720560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.720522 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.720766 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.720677 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.720766 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.720746 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.720898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.720776 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc5rc\" (UniqueName: \"kubernetes.io/projected/3feb82c9-2c7d-46aa-873d-9db55a56ad68-kube-api-access-xc5rc\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.720898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.720815 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.720898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.720861 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.721031 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.720908 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.721031 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.720958 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.721031 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.720979 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.726050 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.726027 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qmwl4\"" Nov 02 20:19:31.733313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.733251 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts"] Nov 02 20:19:31.821631 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.821593 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.821631 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.821637 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.821911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.821664 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.821911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.821694 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.821911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.821723 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.821911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.821777 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.821911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.821868 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.821911 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.821907 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.822242 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.821935 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc5rc\" (UniqueName: \"kubernetes.io/projected/3feb82c9-2c7d-46aa-873d-9db55a56ad68-kube-api-access-xc5rc\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.822242 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.822122 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.822242 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.822132 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.822498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.822415 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.822498 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.822445 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.822816 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.822795 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.824579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.824557 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.824742 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.824724 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.846931 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:19:31.846897 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178183e7_e489_4178_99c3_5309122770e1.slice/crio-9d8647e5f64914e02b52d2c153d264f2cbdf1ad230e191a498d896af8eb595bb WatchSource:0}: Error finding container 9d8647e5f64914e02b52d2c153d264f2cbdf1ad230e191a498d896af8eb595bb: Status 404 returned error can't find the container with id 9d8647e5f64914e02b52d2c153d264f2cbdf1ad230e191a498d896af8eb595bb Nov 02 20:19:31.848441 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.848413 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f"] Nov 02 20:19:31.879805 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.879774 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3feb82c9-2c7d-46aa-873d-9db55a56ad68-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.879943 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.879863 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc5rc\" (UniqueName: \"kubernetes.io/projected/3feb82c9-2c7d-46aa-873d-9db55a56ad68-kube-api-access-xc5rc\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-w2gts\" (UID: \"3feb82c9-2c7d-46aa-873d-9db55a56ad68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:31.948705 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:31.948669 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:32.363891 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:32.363803 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" event={"ID":"178183e7-e489-4178-99c3-5309122770e1","Type":"ContainerStarted","Data":"890972bde7ae35db04b3fc217511ed72ed950c5fd3370579024b65ae1cdb9f6a"} Nov 02 20:19:32.363891 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:32.363857 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" event={"ID":"178183e7-e489-4178-99c3-5309122770e1","Type":"ContainerStarted","Data":"9d8647e5f64914e02b52d2c153d264f2cbdf1ad230e191a498d896af8eb595bb"} Nov 02 20:19:32.375595 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:19:32.375560 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3feb82c9_2c7d_46aa_873d_9db55a56ad68.slice/crio-979802a460315b059805f1699b9e99243c9beaa9d38a0a15c8b2f16cfc819d4b WatchSource:0}: Error finding container 979802a460315b059805f1699b9e99243c9beaa9d38a0a15c8b2f16cfc819d4b: Status 404 returned error can't find the container with id 979802a460315b059805f1699b9e99243c9beaa9d38a0a15c8b2f16cfc819d4b Nov 02 20:19:32.377898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:32.377868 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts"] Nov 02 20:19:32.377995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:32.377911 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:19:32.378072 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:32.377993 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:19:32.378072 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:32.378037 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:19:32.480427 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:32.480357 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" podStartSLOduration=1.480341573 podStartE2EDuration="1.480341573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:32.479016435 +0000 UTC m=+3030.821405516" watchObservedRunningTime="2026-11-02 20:19:32.480341573 +0000 UTC m=+3030.822730653" Nov 02 20:19:33.370484 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:33.370431 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" event={"ID":"3feb82c9-2c7d-46aa-873d-9db55a56ad68","Type":"ContainerStarted","Data":"b83a5c59a1c0bf7347c9db2b2302eadb6492976628060c751b181dd66fc81f2c"} Nov 02 20:19:33.370948 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:33.370520 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" event={"ID":"3feb82c9-2c7d-46aa-873d-9db55a56ad68","Type":"ContainerStarted","Data":"979802a460315b059805f1699b9e99243c9beaa9d38a0a15c8b2f16cfc819d4b"} Nov 02 20:19:33.371966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:33.371943 5850 generic.go:356] "Generic (PLEG): container finished" podID="178183e7-e489-4178-99c3-5309122770e1" containerID="890972bde7ae35db04b3fc217511ed72ed950c5fd3370579024b65ae1cdb9f6a" exitCode=0 Nov 02 20:19:33.372047 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:33.372024 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" event={"ID":"178183e7-e489-4178-99c3-5309122770e1","Type":"ContainerDied","Data":"890972bde7ae35db04b3fc217511ed72ed950c5fd3370579024b65ae1cdb9f6a"} Nov 02 20:19:33.592020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:33.591961 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podStartSLOduration=2.591945727 podStartE2EDuration="2.591945727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:33.540399133 +0000 UTC m=+3031.882788214" watchObservedRunningTime="2026-11-02 20:19:33.591945727 +0000 UTC m=+3031.934334807" Nov 02 20:19:33.949538 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:33.949506 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:33.951084 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:33.951056 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w2gts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:19:33.951207 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:33.951108 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podUID="3feb82c9-2c7d-46aa-873d-9db55a56ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 20:19:34.555308 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.555281 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:34.649201 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.649096 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/178183e7-e489-4178-99c3-5309122770e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178183e7-e489-4178-99c3-5309122770e1-maas-api-xks-ca\") pod \"178183e7-e489-4178-99c3-5309122770e1\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " Nov 02 20:19:34.649392 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.649236 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178183e7-e489-4178-99c3-5309122770e1-kube-api-access-87qkz\" (UniqueName: \"kubernetes.io/projected/178183e7-e489-4178-99c3-5309122770e1-kube-api-access-87qkz\") pod \"178183e7-e489-4178-99c3-5309122770e1\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " Nov 02 20:19:34.649392 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.649346 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178183e7-e489-4178-99c3-5309122770e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178183e7-e489-4178-99c3-5309122770e1-maas-api-openshift-ca\") pod \"178183e7-e489-4178-99c3-5309122770e1\" (UID: \"178183e7-e489-4178-99c3-5309122770e1\") " Nov 02 20:19:34.649690 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.649662 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178183e7-e489-4178-99c3-5309122770e1-maas-api-openshift-ca" pod "178183e7-e489-4178-99c3-5309122770e1" (UID: "178183e7-e489-4178-99c3-5309122770e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:34.651369 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.651342 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178183e7-e489-4178-99c3-5309122770e1-maas-api-xks-ca" pod "178183e7-e489-4178-99c3-5309122770e1" (UID: "178183e7-e489-4178-99c3-5309122770e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:19:34.651457 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.651422 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178183e7-e489-4178-99c3-5309122770e1-kube-api-access-87qkz" pod "178183e7-e489-4178-99c3-5309122770e1" (UID: "178183e7-e489-4178-99c3-5309122770e1"). InnerVolumeSpecName "kube-api-access-87qkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:34.751169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.751131 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87qkz\" (UniqueName: \"kubernetes.io/projected/178183e7-e489-4178-99c3-5309122770e1-kube-api-access-87qkz\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:34.751169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.751164 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178183e7-e489-4178-99c3-5309122770e1-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:34.751169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.751174 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178183e7-e489-4178-99c3-5309122770e1-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:34.949778 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.949685 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w2gts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:19:34.949778 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:34.949745 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podUID="3feb82c9-2c7d-46aa-873d-9db55a56ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 20:19:35.383174 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:35.383141 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" Nov 02 20:19:35.383395 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:35.383176 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f" event={"ID":"178183e7-e489-4178-99c3-5309122770e1","Type":"ContainerDied","Data":"9d8647e5f64914e02b52d2c153d264f2cbdf1ad230e191a498d896af8eb595bb"} Nov 02 20:19:35.383395 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:35.383231 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d8647e5f64914e02b52d2c153d264f2cbdf1ad230e191a498d896af8eb595bb" Nov 02 20:19:35.949902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:35.949855 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w2gts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:19:35.950338 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:35.949926 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podUID="3feb82c9-2c7d-46aa-873d-9db55a56ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 20:19:36.949586 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:36.949551 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w2gts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:19:36.949794 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:36.949614 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podUID="3feb82c9-2c7d-46aa-873d-9db55a56ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 20:19:37.949785 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:37.949743 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w2gts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:19:37.950242 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:37.949813 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podUID="3feb82c9-2c7d-46aa-873d-9db55a56ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 20:19:38.949810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:38.949768 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w2gts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:19:38.950234 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:38.949833 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podUID="3feb82c9-2c7d-46aa-873d-9db55a56ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 20:19:39.949078 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:39.949033 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w2gts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:19:39.949274 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:39.949106 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podUID="3feb82c9-2c7d-46aa-873d-9db55a56ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 20:19:41.989657 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:41.989611 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w2gts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:19:41.990165 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:41.989672 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podUID="3feb82c9-2c7d-46aa-873d-9db55a56ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:19:42.829455 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:42.829415 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:42.829852 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:42.829820 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:42.830616 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:42.830596 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" Nov 02 20:19:43.660830 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:43.660799 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:44.761323 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:19:44.761279 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:19:46.191752 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:46.191718 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:46.890997 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:19:46.890958 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:14.760828 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:20:14.760749 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:20:17.210625 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:17.210593 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:20.604199 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:20.604158 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:44.760591 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:20:44.760546 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:20:49.854131 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.854086 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg"] Nov 02 20:20:49.854703 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.854683 5850 cpu_manager.go:414] "containerMap: removing container" podUID="178183e7-e489-4178-99c3-5309122770e1" containerName="revoke-keys" Nov 02 20:20:49.854784 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.854706 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178183e7-e489-4178-99c3-5309122770e1" containerName="revoke-keys" Nov 02 20:20:49.854870 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.854857 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="178183e7-e489-4178-99c3-5309122770e1" containerName="revoke-keys" Nov 02 20:20:49.858085 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.858064 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:49.913971 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.913936 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg"] Nov 02 20:20:49.963436 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.963397 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-965xg\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:49.963436 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.963455 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-965xg\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:49.963759 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.963576 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:20:49.963759 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.963670 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sbq\" (UniqueName: \"kubernetes.io/projected/2a8c888a-390b-4ea3-ab52-7a8672c6942d-kube-api-access-l4sbq\") pod \"maas-api-revoke-sub-c1e271a9-965xg\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:49.963759 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:49.963684 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:20:50.064849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:50.064804 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sbq\" (UniqueName: \"kubernetes.io/projected/2a8c888a-390b-4ea3-ab52-7a8672c6942d-kube-api-access-l4sbq\") pod \"maas-api-revoke-sub-c1e271a9-965xg\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:50.064849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:50.064848 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-965xg\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:50.065090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:50.064888 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-965xg\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:50.065620 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:50.065594 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-965xg\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:50.067283 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:50.067261 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-965xg\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:50.122896 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:50.122863 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sbq\" (UniqueName: \"kubernetes.io/projected/2a8c888a-390b-4ea3-ab52-7a8672c6942d-kube-api-access-l4sbq\") pod \"maas-api-revoke-sub-c1e271a9-965xg\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:50.170964 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:50.170918 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:50.537761 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:20:50.537725 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8c888a_390b_4ea3_ab52_7a8672c6942d.slice/crio-d5b8595773b8b03d5213461f4a2819e3b3b75be6fb41f8885b61852b46b4d7f0 WatchSource:0}: Error finding container d5b8595773b8b03d5213461f4a2819e3b3b75be6fb41f8885b61852b46b4d7f0: Status 404 returned error can't find the container with id d5b8595773b8b03d5213461f4a2819e3b3b75be6fb41f8885b61852b46b4d7f0 Nov 02 20:20:50.539644 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:50.539622 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg"] Nov 02 20:20:50.737241 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:50.737204 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" event={"ID":"2a8c888a-390b-4ea3-ab52-7a8672c6942d","Type":"ContainerStarted","Data":"d5b8595773b8b03d5213461f4a2819e3b3b75be6fb41f8885b61852b46b4d7f0"} Nov 02 20:20:51.743338 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:51.743251 5850 generic.go:356] "Generic (PLEG): container finished" podID="2a8c888a-390b-4ea3-ab52-7a8672c6942d" containerID="924007d7d45c98b5dea74b68aa68086107c562a872c2f1ce2cd647eb722bdd28" exitCode=0 Nov 02 20:20:51.743810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:51.743336 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" event={"ID":"2a8c888a-390b-4ea3-ab52-7a8672c6942d","Type":"ContainerDied","Data":"924007d7d45c98b5dea74b68aa68086107c562a872c2f1ce2cd647eb722bdd28"} Nov 02 20:20:52.881279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:52.881252 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:52.991934 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:52.991895 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8c888a-390b-4ea3-ab52-7a8672c6942d-kube-api-access-l4sbq\" (UniqueName: \"kubernetes.io/projected/2a8c888a-390b-4ea3-ab52-7a8672c6942d-kube-api-access-l4sbq\") pod \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " Nov 02 20:20:52.991934 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:52.991945 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-xks-ca\") pod \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " Nov 02 20:20:52.992172 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:52.991989 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-openshift-ca\") pod \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\" (UID: \"2a8c888a-390b-4ea3-ab52-7a8672c6942d\") " Nov 02 20:20:52.992440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:52.992419 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-openshift-ca" pod "2a8c888a-390b-4ea3-ab52-7a8672c6942d" (UID: "2a8c888a-390b-4ea3-ab52-7a8672c6942d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:52.994207 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:52.994184 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8c888a-390b-4ea3-ab52-7a8672c6942d-kube-api-access-l4sbq" pod "2a8c888a-390b-4ea3-ab52-7a8672c6942d" (UID: "2a8c888a-390b-4ea3-ab52-7a8672c6942d"). InnerVolumeSpecName "kube-api-access-l4sbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:52.994343 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:52.994207 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-xks-ca" pod "2a8c888a-390b-4ea3-ab52-7a8672c6942d" (UID: "2a8c888a-390b-4ea3-ab52-7a8672c6942d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:20:53.093289 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:53.093249 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4sbq\" (UniqueName: \"kubernetes.io/projected/2a8c888a-390b-4ea3-ab52-7a8672c6942d-kube-api-access-l4sbq\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:53.093469 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:53.093304 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:53.093469 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:53.093315 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c888a-390b-4ea3-ab52-7a8672c6942d-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:53.753858 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:53.753822 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" event={"ID":"2a8c888a-390b-4ea3-ab52-7a8672c6942d","Type":"ContainerDied","Data":"d5b8595773b8b03d5213461f4a2819e3b3b75be6fb41f8885b61852b46b4d7f0"} Nov 02 20:20:53.753858 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:53.753852 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg" Nov 02 20:20:53.754077 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:53.753897 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5b8595773b8b03d5213461f4a2819e3b3b75be6fb41f8885b61852b46b4d7f0" Nov 02 20:20:58.960313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:20:58.960276 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:02.248459 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.248415 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5"] Nov 02 20:21:02.248916 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.248844 5850 cpu_manager.go:414] "containerMap: removing container" podUID="2a8c888a-390b-4ea3-ab52-7a8672c6942d" containerName="revoke-keys" Nov 02 20:21:02.248916 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.248857 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8c888a-390b-4ea3-ab52-7a8672c6942d" containerName="revoke-keys" Nov 02 20:21:02.249079 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.248928 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8c888a-390b-4ea3-ab52-7a8672c6942d" containerName="revoke-keys" Nov 02 20:21:02.252105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.252084 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.354451 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.354415 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5"] Nov 02 20:21:02.386730 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.386687 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.386913 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.386740 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.386913 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.386827 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.386913 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.386894 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.387027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.386939 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.387027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.386962 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.387096 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.387076 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.387146 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.387126 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.387183 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.387166 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dbph\" (UniqueName: \"kubernetes.io/projected/1d8c479b-a392-4f75-9e43-c25fe3b73ada-kube-api-access-9dbph\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.488399 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488353 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.488399 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488399 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.488695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488431 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dbph\" (UniqueName: \"kubernetes.io/projected/1d8c479b-a392-4f75-9e43-c25fe3b73ada-kube-api-access-9dbph\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.488695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488506 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.488695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488545 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.488695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488570 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.488695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488606 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.488695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488663 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.488695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488689 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.489075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488864 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.489075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.488950 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.489197 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.489159 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.489197 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.489181 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.489457 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.489430 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.491509 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.491464 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.491832 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.491805 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.547652 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.547570 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d8c479b-a392-4f75-9e43-c25fe3b73ada-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.547805 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.547699 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dbph\" (UniqueName: \"kubernetes.io/projected/1d8c479b-a392-4f75-9e43-c25fe3b73ada-kube-api-access-9dbph\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-k56z5\" (UID: \"1d8c479b-a392-4f75-9e43-c25fe3b73ada\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.565599 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.565560 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:02.987770 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:02.987733 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5"] Nov 02 20:21:03.024217 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:21:03.024184 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8c479b_a392_4f75_9e43_c25fe3b73ada.slice/crio-078021a1a265092c0684c718cd2642f30e9f641c012a38640837165a41baecec WatchSource:0}: Error finding container 078021a1a265092c0684c718cd2642f30e9f641c012a38640837165a41baecec: Status 404 returned error can't find the container with id 078021a1a265092c0684c718cd2642f30e9f641c012a38640837165a41baecec Nov 02 20:21:03.026330 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:03.026293 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:21:03.026407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:03.026368 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:21:03.026407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:03.026401 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:21:03.804482 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:03.804436 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" event={"ID":"1d8c479b-a392-4f75-9e43-c25fe3b73ada","Type":"ContainerStarted","Data":"f7c11d8fe393ec8d91a569f12614b4e3d1ee8bdcf3667d7cf1872c10714413cc"} Nov 02 20:21:03.804888 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:03.804503 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" event={"ID":"1d8c479b-a392-4f75-9e43-c25fe3b73ada","Type":"ContainerStarted","Data":"078021a1a265092c0684c718cd2642f30e9f641c012a38640837165a41baecec"} Nov 02 20:21:04.566018 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:04.565973 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:04.567531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:04.567494 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-k56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 20:21:04.567661 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:04.567548 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podUID="1d8c479b-a392-4f75-9e43-c25fe3b73ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 20:21:05.566000 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:05.565961 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-k56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 20:21:05.566447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:05.566031 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podUID="1d8c479b-a392-4f75-9e43-c25fe3b73ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 20:21:05.688531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:05.688496 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:05.833408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:05.833306 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:06.566943 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:06.566903 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-k56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 20:21:06.567377 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:06.566976 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podUID="1d8c479b-a392-4f75-9e43-c25fe3b73ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 20:21:07.566342 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:07.566305 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-k56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 20:21:07.566567 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:07.566364 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podUID="1d8c479b-a392-4f75-9e43-c25fe3b73ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 20:21:08.566564 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:08.566517 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-k56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 20:21:08.567129 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:08.566598 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podUID="1d8c479b-a392-4f75-9e43-c25fe3b73ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 20:21:09.566924 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:09.566873 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-k56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 20:21:09.567360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:09.566969 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podUID="1d8c479b-a392-4f75-9e43-c25fe3b73ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 20:21:11.567635 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:11.567587 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-k56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:21:11.568141 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:11.567658 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podUID="1d8c479b-a392-4f75-9e43-c25fe3b73ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:21:12.608695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:12.608641 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-k56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:21:12.609229 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:12.608703 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podUID="1d8c479b-a392-4f75-9e43-c25fe3b73ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:21:13.347918 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:13.347881 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:13.348296 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:13.348272 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:13.349056 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:13.349034 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" Nov 02 20:21:13.598792 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:13.598680 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podStartSLOduration=11.598660555 podStartE2EDuration="11.598660555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:04.044285019 +0000 UTC m=+3122.386674099" watchObservedRunningTime="2026-11-02 20:21:13.598660555 +0000 UTC m=+3131.941049635" Nov 02 20:21:14.760918 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:21:14.760877 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:21:17.358006 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.357963 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp"] Nov 02 20:21:17.361656 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.361636 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.468608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.468564 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.468608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.468624 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.468906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.468683 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.468906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.468709 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.468906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.468758 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.468906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.468795 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5j5\" (UniqueName: \"kubernetes.io/projected/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-kube-api-access-wn5j5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.468906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.468854 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.469188 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.468916 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.469188 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.468946 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.469872 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.469843 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp"] Nov 02 20:21:17.569863 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.569821 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.569863 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.569869 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5j5\" (UniqueName: \"kubernetes.io/projected/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-kube-api-access-wn5j5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570158 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.569990 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570158 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570039 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570158 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570060 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570158 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570113 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570158 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570134 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570186 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570212 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570611 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570572 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570686 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570607 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570698 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570800 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570734 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.570949 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.570926 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.572421 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.572391 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.572589 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.572525 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.628530 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.628496 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.628680 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.628575 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5j5\" (UniqueName: \"kubernetes.io/projected/ac5dc2a4-1609-4d43-b359-19adbb7aaeb8-kube-api-access-wn5j5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hszrp\" (UID: \"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:17.674408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:17.674370 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:18.099362 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:18.099327 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp"] Nov 02 20:21:18.099627 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:21:18.099600 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5dc2a4_1609_4d43_b359_19adbb7aaeb8.slice/crio-9b0e574791a0111c5a856bf17f2d608770d448808a9e296233791d29cb8091e0 WatchSource:0}: Error finding container 9b0e574791a0111c5a856bf17f2d608770d448808a9e296233791d29cb8091e0: Status 404 returned error can't find the container with id 9b0e574791a0111c5a856bf17f2d608770d448808a9e296233791d29cb8091e0 Nov 02 20:21:18.102023 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:18.101980 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:21:18.102112 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:18.102074 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:21:18.102169 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:18.102117 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:21:18.904131 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:18.904098 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" event={"ID":"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8","Type":"ContainerStarted","Data":"070108cfd27969277e3cf80084e33a94975a03d3cf67d13ad61565bcd875a832"} Nov 02 20:21:18.904564 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:18.904147 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" event={"ID":"ac5dc2a4-1609-4d43-b359-19adbb7aaeb8","Type":"ContainerStarted","Data":"9b0e574791a0111c5a856bf17f2d608770d448808a9e296233791d29cb8091e0"} Nov 02 20:21:19.090871 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:19.090814 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podStartSLOduration=2.090799613 podStartE2EDuration="2.090799613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:19.089802179 +0000 UTC m=+3137.432191259" watchObservedRunningTime="2026-11-02 20:21:19.090799613 +0000 UTC m=+3137.433188690" Nov 02 20:21:19.675011 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:19.674969 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:19.676629 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:19.676604 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hszrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:21:19.676772 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:19.676654 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podUID="ac5dc2a4-1609-4d43-b359-19adbb7aaeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 20:21:20.675380 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:20.675344 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hszrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:21:20.675868 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:20.675401 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podUID="ac5dc2a4-1609-4d43-b359-19adbb7aaeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 20:21:20.864791 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:20.864753 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:21.250276 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.250239 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw"] Nov 02 20:21:21.254046 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.254025 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.319500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.312432 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:21:21.319500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.319075 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfcg5\" (UniqueName: \"kubernetes.io/projected/cddf633d-93a4-4c5d-98df-9c6113d82271-kube-api-access-zfcg5\") pod \"maas-api-revoke-sub-2e1f5b5f-dcmrw\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.319500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.319197 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dcmrw\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.319500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.319267 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dcmrw\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.342148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.342121 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:21.363551 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.363514 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:21:21.365113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.365086 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw"] Nov 02 20:21:21.428820 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.428055 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfcg5\" (UniqueName: \"kubernetes.io/projected/cddf633d-93a4-4c5d-98df-9c6113d82271-kube-api-access-zfcg5\") pod \"maas-api-revoke-sub-2e1f5b5f-dcmrw\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.428820 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.428125 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dcmrw\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.428820 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.428188 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dcmrw\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.430291 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.430259 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dcmrw\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.433223 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.433198 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dcmrw\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.488284 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.488251 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfcg5\" (UniqueName: \"kubernetes.io/projected/cddf633d-93a4-4c5d-98df-9c6113d82271-kube-api-access-zfcg5\") pod \"maas-api-revoke-sub-2e1f5b5f-dcmrw\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.568121 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.568036 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:21.675155 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.675119 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hszrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:21:21.675353 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.675181 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podUID="ac5dc2a4-1609-4d43-b359-19adbb7aaeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 20:21:21.965506 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:21:21.961716 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddf633d_93a4_4c5d_98df_9c6113d82271.slice/crio-235403efe32792941ecc6480f41ff646b05a79c6a12dc6b8a50912165ef425f9 WatchSource:0}: Error finding container 235403efe32792941ecc6480f41ff646b05a79c6a12dc6b8a50912165ef425f9: Status 404 returned error can't find the container with id 235403efe32792941ecc6480f41ff646b05a79c6a12dc6b8a50912165ef425f9 Nov 02 20:21:21.965506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:21.962339 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw"] Nov 02 20:21:22.674865 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:22.674823 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hszrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:21:22.675033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:22.674887 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podUID="ac5dc2a4-1609-4d43-b359-19adbb7aaeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 20:21:22.949903 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:22.949807 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" event={"ID":"cddf633d-93a4-4c5d-98df-9c6113d82271","Type":"ContainerStarted","Data":"9eedfa1cf1ba4eeba48adff9044cf6f77c75bf20a3744f569add00fdce160109"} Nov 02 20:21:22.949903 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:22.949895 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" event={"ID":"cddf633d-93a4-4c5d-98df-9c6113d82271","Type":"ContainerStarted","Data":"235403efe32792941ecc6480f41ff646b05a79c6a12dc6b8a50912165ef425f9"} Nov 02 20:21:23.065909 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:23.065843 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" podStartSLOduration=2.065830956 podStartE2EDuration="2.065830956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:23.065117449 +0000 UTC m=+3141.407506528" watchObservedRunningTime="2026-11-02 20:21:23.065830956 +0000 UTC m=+3141.408220035" Nov 02 20:21:23.674916 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:23.674873 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hszrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:21:23.675140 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:23.674936 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podUID="ac5dc2a4-1609-4d43-b359-19adbb7aaeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 20:21:23.956464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:23.956377 5850 generic.go:356] "Generic (PLEG): container finished" podID="cddf633d-93a4-4c5d-98df-9c6113d82271" containerID="9eedfa1cf1ba4eeba48adff9044cf6f77c75bf20a3744f569add00fdce160109" exitCode=0 Nov 02 20:21:23.956652 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:23.956467 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" event={"ID":"cddf633d-93a4-4c5d-98df-9c6113d82271","Type":"ContainerDied","Data":"9eedfa1cf1ba4eeba48adff9044cf6f77c75bf20a3744f569add00fdce160109"} Nov 02 20:21:24.675089 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:24.675049 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hszrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:21:24.675510 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:24.675121 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podUID="ac5dc2a4-1609-4d43-b359-19adbb7aaeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 20:21:25.106553 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.106524 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:25.167669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.167620 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cddf633d-93a4-4c5d-98df-9c6113d82271-kube-api-access-zfcg5\" (UniqueName: \"kubernetes.io/projected/cddf633d-93a4-4c5d-98df-9c6113d82271-kube-api-access-zfcg5\") pod \"cddf633d-93a4-4c5d-98df-9c6113d82271\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " Nov 02 20:21:25.167851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.167714 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-openshift-ca\") pod \"cddf633d-93a4-4c5d-98df-9c6113d82271\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " Nov 02 20:21:25.167851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.167755 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-xks-ca\") pod \"cddf633d-93a4-4c5d-98df-9c6113d82271\" (UID: \"cddf633d-93a4-4c5d-98df-9c6113d82271\") " Nov 02 20:21:25.168120 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.168091 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-openshift-ca" pod "cddf633d-93a4-4c5d-98df-9c6113d82271" (UID: "cddf633d-93a4-4c5d-98df-9c6113d82271"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:25.168216 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.168200 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:25.169950 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.169928 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-xks-ca" pod "cddf633d-93a4-4c5d-98df-9c6113d82271" (UID: "cddf633d-93a4-4c5d-98df-9c6113d82271"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:21:25.170089 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.170072 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddf633d-93a4-4c5d-98df-9c6113d82271-kube-api-access-zfcg5" pod "cddf633d-93a4-4c5d-98df-9c6113d82271" (UID: "cddf633d-93a4-4c5d-98df-9c6113d82271"). InnerVolumeSpecName "kube-api-access-zfcg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:25.269608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.269523 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfcg5\" (UniqueName: \"kubernetes.io/projected/cddf633d-93a4-4c5d-98df-9c6113d82271-kube-api-access-zfcg5\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:25.269608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.269553 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddf633d-93a4-4c5d-98df-9c6113d82271-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:25.675715 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.675676 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hszrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:21:25.676133 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.675736 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podUID="ac5dc2a4-1609-4d43-b359-19adbb7aaeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 20:21:25.969351 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.969250 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" Nov 02 20:21:25.969351 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.969278 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw" event={"ID":"cddf633d-93a4-4c5d-98df-9c6113d82271","Type":"ContainerDied","Data":"235403efe32792941ecc6480f41ff646b05a79c6a12dc6b8a50912165ef425f9"} Nov 02 20:21:25.969351 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:25.969331 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="235403efe32792941ecc6480f41ff646b05a79c6a12dc6b8a50912165ef425f9" Nov 02 20:21:26.679112 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:26.679083 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:26.679541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:26.679517 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:26.680185 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:26.680165 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" Nov 02 20:21:33.543063 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.543019 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w"] Nov 02 20:21:33.543758 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.543735 5850 cpu_manager.go:414] "containerMap: removing container" podUID="cddf633d-93a4-4c5d-98df-9c6113d82271" containerName="revoke-keys" Nov 02 20:21:33.543811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.543761 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cddf633d-93a4-4c5d-98df-9c6113d82271" containerName="revoke-keys" Nov 02 20:21:33.543928 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.543916 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="cddf633d-93a4-4c5d-98df-9c6113d82271" containerName="revoke-keys" Nov 02 20:21:33.547664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.547645 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.603369 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.603327 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w"] Nov 02 20:21:33.652505 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.652417 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:21:33.652703 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.652644 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:21:33.654055 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.654028 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb65t\" (UniqueName: \"kubernetes.io/projected/e22c9791-218a-4482-ac79-9122fc7e3b05-kube-api-access-sb65t\") pod \"maas-api-revoke-sub-ec843018-2w56w\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.654171 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.654153 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2w56w\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.654221 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.654210 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2w56w\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.755163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.755119 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2w56w\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.755363 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.755195 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2w56w\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.755363 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.755265 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb65t\" (UniqueName: \"kubernetes.io/projected/e22c9791-218a-4482-ac79-9122fc7e3b05-kube-api-access-sb65t\") pod \"maas-api-revoke-sub-ec843018-2w56w\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.755885 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.755855 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2w56w\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.757749 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.757729 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2w56w\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.814259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.814186 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb65t\" (UniqueName: \"kubernetes.io/projected/e22c9791-218a-4482-ac79-9122fc7e3b05-kube-api-access-sb65t\") pod \"maas-api-revoke-sub-ec843018-2w56w\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:33.861171 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:33.861125 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:34.231418 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:21:34.231381 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22c9791_218a_4482_ac79_9122fc7e3b05.slice/crio-9faea22b28a058822f91690856c0bde11259e0c9ff634a0d18d373e5d9ce1106 WatchSource:0}: Error finding container 9faea22b28a058822f91690856c0bde11259e0c9ff634a0d18d373e5d9ce1106: Status 404 returned error can't find the container with id 9faea22b28a058822f91690856c0bde11259e0c9ff634a0d18d373e5d9ce1106 Nov 02 20:21:34.233105 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:34.233068 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w"] Nov 02 20:21:35.016602 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:35.016563 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" event={"ID":"e22c9791-218a-4482-ac79-9122fc7e3b05","Type":"ContainerStarted","Data":"6be66a3b254dd9f8718fc0049e61e90d28a4bfdc32ce0e494f1fc23d16274157"} Nov 02 20:21:35.017001 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:35.016625 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" event={"ID":"e22c9791-218a-4482-ac79-9122fc7e3b05","Type":"ContainerStarted","Data":"9faea22b28a058822f91690856c0bde11259e0c9ff634a0d18d373e5d9ce1106"} Nov 02 20:21:35.133391 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:35.133336 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" podStartSLOduration=2.133321285 podStartE2EDuration="2.133321285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:35.131729834 +0000 UTC m=+3153.474118913" watchObservedRunningTime="2026-11-02 20:21:35.133321285 +0000 UTC m=+3153.475710364" Nov 02 20:21:36.023196 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:36.023165 5850 generic.go:356] "Generic (PLEG): container finished" podID="e22c9791-218a-4482-ac79-9122fc7e3b05" containerID="6be66a3b254dd9f8718fc0049e61e90d28a4bfdc32ce0e494f1fc23d16274157" exitCode=0 Nov 02 20:21:36.023667 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:36.023255 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" event={"ID":"e22c9791-218a-4482-ac79-9122fc7e3b05","Type":"ContainerDied","Data":"6be66a3b254dd9f8718fc0049e61e90d28a4bfdc32ce0e494f1fc23d16274157"} Nov 02 20:21:37.165439 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.165412 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:37.292946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.292855 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-openshift-ca\") pod \"e22c9791-218a-4482-ac79-9122fc7e3b05\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " Nov 02 20:21:37.292946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.292931 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-xks-ca\") pod \"e22c9791-218a-4482-ac79-9122fc7e3b05\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " Nov 02 20:21:37.293176 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.292970 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22c9791-218a-4482-ac79-9122fc7e3b05-kube-api-access-sb65t\" (UniqueName: \"kubernetes.io/projected/e22c9791-218a-4482-ac79-9122fc7e3b05-kube-api-access-sb65t\") pod \"e22c9791-218a-4482-ac79-9122fc7e3b05\" (UID: \"e22c9791-218a-4482-ac79-9122fc7e3b05\") " Nov 02 20:21:37.293329 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.293300 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-openshift-ca" pod "e22c9791-218a-4482-ac79-9122fc7e3b05" (UID: "e22c9791-218a-4482-ac79-9122fc7e3b05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:37.295127 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.295099 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-xks-ca" pod "e22c9791-218a-4482-ac79-9122fc7e3b05" (UID: "e22c9791-218a-4482-ac79-9122fc7e3b05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:21:37.295243 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.295152 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22c9791-218a-4482-ac79-9122fc7e3b05-kube-api-access-sb65t" pod "e22c9791-218a-4482-ac79-9122fc7e3b05" (UID: "e22c9791-218a-4482-ac79-9122fc7e3b05"). InnerVolumeSpecName "kube-api-access-sb65t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:37.393828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.393795 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:37.393828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.393825 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22c9791-218a-4482-ac79-9122fc7e3b05-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:37.393828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:37.393835 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb65t\" (UniqueName: \"kubernetes.io/projected/e22c9791-218a-4482-ac79-9122fc7e3b05-kube-api-access-sb65t\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:38.036663 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:38.036623 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" Nov 02 20:21:38.036849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:38.036621 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w" event={"ID":"e22c9791-218a-4482-ac79-9122fc7e3b05","Type":"ContainerDied","Data":"9faea22b28a058822f91690856c0bde11259e0c9ff634a0d18d373e5d9ce1106"} Nov 02 20:21:38.036849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:21:38.036750 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9faea22b28a058822f91690856c0bde11259e0c9ff634a0d18d373e5d9ce1106" Nov 02 20:21:44.761189 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:21:44.761105 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:22:02.701096 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.701057 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl"] Nov 02 20:22:02.701583 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.701519 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e22c9791-218a-4482-ac79-9122fc7e3b05" containerName="revoke-keys" Nov 02 20:22:02.701583 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.701531 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22c9791-218a-4482-ac79-9122fc7e3b05" containerName="revoke-keys" Nov 02 20:22:02.701696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.701598 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22c9791-218a-4482-ac79-9122fc7e3b05" containerName="revoke-keys" Nov 02 20:22:02.704716 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.704698 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.763892 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.763847 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.764076 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.763907 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.764076 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.763965 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.764076 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.764056 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.764229 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.764091 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.764229 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.764112 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.764229 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.764135 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.764447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.764276 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.764447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.764333 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrcx4\" (UniqueName: \"kubernetes.io/projected/6222ddb5-cd09-43db-82f2-01c85b1e63d7-kube-api-access-mrcx4\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.801095 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.801058 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl"] Nov 02 20:22:02.865660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.865615 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrcx4\" (UniqueName: \"kubernetes.io/projected/6222ddb5-cd09-43db-82f2-01c85b1e63d7-kube-api-access-mrcx4\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.865915 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.865885 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866053 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.865940 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866053 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.865995 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866182 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866082 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866182 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866127 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866182 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866156 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866190 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866306 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866458 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866365 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866577 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866457 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866643 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866615 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866701 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866663 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.866798 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.866762 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.868619 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.868600 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.868771 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.868752 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.924548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.924515 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6222ddb5-cd09-43db-82f2-01c85b1e63d7-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:02.924708 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:02.924685 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrcx4\" (UniqueName: \"kubernetes.io/projected/6222ddb5-cd09-43db-82f2-01c85b1e63d7-kube-api-access-mrcx4\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d2chl\" (UID: \"6222ddb5-cd09-43db-82f2-01c85b1e63d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:03.021726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:03.021641 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:03.444117 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:03.444082 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl"] Nov 02 20:22:03.479454 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:22:03.479423 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6222ddb5_cd09_43db_82f2_01c85b1e63d7.slice/crio-90bcad44b0252ab0ce7d281ec464dae586f766af98378e91936e5ba7c7d8cd24 WatchSource:0}: Error finding container 90bcad44b0252ab0ce7d281ec464dae586f766af98378e91936e5ba7c7d8cd24: Status 404 returned error can't find the container with id 90bcad44b0252ab0ce7d281ec464dae586f766af98378e91936e5ba7c7d8cd24 Nov 02 20:22:03.481718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:03.481682 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:22:03.481798 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:03.481755 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:22:03.481798 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:03.481785 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:22:04.201324 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:04.201280 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" event={"ID":"6222ddb5-cd09-43db-82f2-01c85b1e63d7","Type":"ContainerStarted","Data":"8cb9e06cbbe1f8e2d7274084e00754799fe4e0c8d5de77181629c12fc8e356d2"} Nov 02 20:22:04.201774 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:04.201336 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" event={"ID":"6222ddb5-cd09-43db-82f2-01c85b1e63d7","Type":"ContainerStarted","Data":"90bcad44b0252ab0ce7d281ec464dae586f766af98378e91936e5ba7c7d8cd24"} Nov 02 20:22:04.719781 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:04.719665 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:05.027435 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:05.027337 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:05.027846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:05.027822 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d2chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:22:05.028021 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:05.027995 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" podUID="6222ddb5-cd09-43db-82f2-01c85b1e63d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 20:22:06.027020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:06.026971 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d2chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:22:06.027676 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:06.027635 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" podUID="6222ddb5-cd09-43db-82f2-01c85b1e63d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 20:22:07.029499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:07.028915 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d2chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:22:07.029499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:07.028981 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" podUID="6222ddb5-cd09-43db-82f2-01c85b1e63d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 20:22:08.026968 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:08.026927 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d2chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:22:08.028107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:08.028073 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" podUID="6222ddb5-cd09-43db-82f2-01c85b1e63d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 20:22:09.027499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:09.026912 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d2chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:22:09.027499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:09.026982 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" podUID="6222ddb5-cd09-43db-82f2-01c85b1e63d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 20:22:10.022819 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:10.022779 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d2chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:22:10.023011 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:10.022839 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" podUID="6222ddb5-cd09-43db-82f2-01c85b1e63d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 20:22:11.022449 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:11.022404 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d2chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:22:11.022907 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:11.022465 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" podUID="6222ddb5-cd09-43db-82f2-01c85b1e63d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 20:22:12.026025 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.025994 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:12.026565 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.026397 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:12.027188 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.027166 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" Nov 02 20:22:12.196332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.196280 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" podStartSLOduration=10.196266707 podStartE2EDuration="10.196266707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:04.394378049 +0000 UTC m=+3182.736767128" watchObservedRunningTime="2026-11-02 20:22:12.196266707 +0000 UTC m=+3190.538655781" Nov 02 20:22:12.268778 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.268746 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs"] Nov 02 20:22:12.272515 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.272464 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.330992 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.330952 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs"] Nov 02 20:22:12.331367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.331333 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:22:12.331516 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.331322 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:22:12.391646 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.391602 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-m7dfs\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.391849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.391702 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hqj\" (UniqueName: \"kubernetes.io/projected/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-kube-api-access-f9hqj\") pod \"maas-api-revoke-sub-046e1343-m7dfs\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.391849 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.391818 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-m7dfs\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.492568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.492530 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-m7dfs\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.492770 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.492602 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hqj\" (UniqueName: \"kubernetes.io/projected/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-kube-api-access-f9hqj\") pod \"maas-api-revoke-sub-046e1343-m7dfs\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.492770 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.492633 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-m7dfs\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.493347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.493322 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-m7dfs\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.495014 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.494988 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-m7dfs\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.550931 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.550855 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hqj\" (UniqueName: \"kubernetes.io/projected/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-kube-api-access-f9hqj\") pod \"maas-api-revoke-sub-046e1343-m7dfs\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.585933 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.585895 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:12.956547 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:22:12.956506 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf06cda2_38e9_4d09_aa3f_e0f705f0aa4d.slice/crio-87170997cb7b9d7eded2370b192ae3c5535561244d869d24620e83655591c361 WatchSource:0}: Error finding container 87170997cb7b9d7eded2370b192ae3c5535561244d869d24620e83655591c361: Status 404 returned error can't find the container with id 87170997cb7b9d7eded2370b192ae3c5535561244d869d24620e83655591c361 Nov 02 20:22:12.956726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:12.956616 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs"] Nov 02 20:22:13.273357 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:13.273316 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" event={"ID":"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d","Type":"ContainerStarted","Data":"16b76553db5995e6af014169ef02054d61b309a0d4513ec982e625a25fa4eeca"} Nov 02 20:22:13.273775 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:13.273384 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" event={"ID":"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d","Type":"ContainerStarted","Data":"87170997cb7b9d7eded2370b192ae3c5535561244d869d24620e83655591c361"} Nov 02 20:22:13.389428 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:13.389369 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" podStartSLOduration=1.389355213 podStartE2EDuration="1.389355213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:13.388788806 +0000 UTC m=+3191.731177909" watchObservedRunningTime="2026-11-02 20:22:13.389355213 +0000 UTC m=+3191.731744289" Nov 02 20:22:14.279464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:14.279426 5850 generic.go:356] "Generic (PLEG): container finished" podID="df06cda2-38e9-4d09-aa3f-e0f705f0aa4d" containerID="16b76553db5995e6af014169ef02054d61b309a0d4513ec982e625a25fa4eeca" exitCode=0 Nov 02 20:22:14.279879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:14.279514 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" event={"ID":"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d","Type":"ContainerDied","Data":"16b76553db5995e6af014169ef02054d61b309a0d4513ec982e625a25fa4eeca"} Nov 02 20:22:14.762135 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:22:14.762096 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:22:15.423855 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.423828 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:15.522225 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.522178 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-openshift-ca\") pod \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " Nov 02 20:22:15.522428 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.522339 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-kube-api-access-f9hqj\" (UniqueName: \"kubernetes.io/projected/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-kube-api-access-f9hqj\") pod \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " Nov 02 20:22:15.522428 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.522367 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-xks-ca\") pod \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\" (UID: \"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d\") " Nov 02 20:22:15.522624 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.522598 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-openshift-ca" pod "df06cda2-38e9-4d09-aa3f-e0f705f0aa4d" (UID: "df06cda2-38e9-4d09-aa3f-e0f705f0aa4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:22:15.524615 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.524590 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-xks-ca" pod "df06cda2-38e9-4d09-aa3f-e0f705f0aa4d" (UID: "df06cda2-38e9-4d09-aa3f-e0f705f0aa4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:22:15.524747 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.524655 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-kube-api-access-f9hqj" pod "df06cda2-38e9-4d09-aa3f-e0f705f0aa4d" (UID: "df06cda2-38e9-4d09-aa3f-e0f705f0aa4d"). InnerVolumeSpecName "kube-api-access-f9hqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:22:15.623644 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.623606 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:15.623644 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.623639 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:15.623644 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:15.623651 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hqj\" (UniqueName: \"kubernetes.io/projected/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d-kube-api-access-f9hqj\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:16.290688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:16.290653 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" Nov 02 20:22:16.290688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:16.290667 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs" event={"ID":"df06cda2-38e9-4d09-aa3f-e0f705f0aa4d","Type":"ContainerDied","Data":"87170997cb7b9d7eded2370b192ae3c5535561244d869d24620e83655591c361"} Nov 02 20:22:16.290906 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:16.290718 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87170997cb7b9d7eded2370b192ae3c5535561244d869d24620e83655591c361" Nov 02 20:22:21.190866 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:21.190835 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:21.552737 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:21.552646 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:22.962891 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:22.962852 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh"] Nov 02 20:22:22.963369 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:22.963333 5850 cpu_manager.go:414] "containerMap: removing container" podUID="df06cda2-38e9-4d09-aa3f-e0f705f0aa4d" containerName="revoke-keys" Nov 02 20:22:22.963369 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:22.963349 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df06cda2-38e9-4d09-aa3f-e0f705f0aa4d" containerName="revoke-keys" Nov 02 20:22:22.963448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:22.963421 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="df06cda2-38e9-4d09-aa3f-e0f705f0aa4d" containerName="revoke-keys" Nov 02 20:22:22.966810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:22.966787 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:22.999936 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:22.999903 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bccsh\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.000138 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:22.999956 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kx8\" (UniqueName: \"kubernetes.io/projected/d3340609-1895-4a4e-8dee-4319e1fecac0-kube-api-access-z8kx8\") pod \"maas-api-revoke-sub-d3fba056-bccsh\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.000138 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.000078 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bccsh\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.023649 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.023609 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh"] Nov 02 20:22:23.072263 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.072224 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:22:23.072450 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.072425 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:22:23.101103 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.101061 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bccsh\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.101302 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.101136 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kx8\" (UniqueName: \"kubernetes.io/projected/d3340609-1895-4a4e-8dee-4319e1fecac0-kube-api-access-z8kx8\") pod \"maas-api-revoke-sub-d3fba056-bccsh\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.101302 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.101186 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bccsh\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.101859 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.101828 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bccsh\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.103695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.103673 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bccsh\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.159646 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.159608 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kx8\" (UniqueName: \"kubernetes.io/projected/d3340609-1895-4a4e-8dee-4319e1fecac0-kube-api-access-z8kx8\") pod \"maas-api-revoke-sub-d3fba056-bccsh\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.283620 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.283506 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:23.652880 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:22:23.652842 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3340609_1895_4a4e_8dee_4319e1fecac0.slice/crio-f698c873d9ef07791e66ebbedbb7264529b6bd51fb8259edd1330f838b7eba1e WatchSource:0}: Error finding container f698c873d9ef07791e66ebbedbb7264529b6bd51fb8259edd1330f838b7eba1e: Status 404 returned error can't find the container with id f698c873d9ef07791e66ebbedbb7264529b6bd51fb8259edd1330f838b7eba1e Nov 02 20:22:23.654359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:23.654332 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh"] Nov 02 20:22:24.330939 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:24.330838 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" event={"ID":"d3340609-1895-4a4e-8dee-4319e1fecac0","Type":"ContainerStarted","Data":"944367e2e0c0b2ce3864374a20e2861a1899e24e428c50539bdb67be41adfe19"} Nov 02 20:22:24.330939 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:24.330901 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" event={"ID":"d3340609-1895-4a4e-8dee-4319e1fecac0","Type":"ContainerStarted","Data":"f698c873d9ef07791e66ebbedbb7264529b6bd51fb8259edd1330f838b7eba1e"} Nov 02 20:22:24.448441 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:24.448380 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" podStartSLOduration=2.448363945 podStartE2EDuration="2.448363945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:24.446673028 +0000 UTC m=+3202.789062109" watchObservedRunningTime="2026-11-02 20:22:24.448363945 +0000 UTC m=+3202.790753025" Nov 02 20:22:25.337637 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:25.337599 5850 generic.go:356] "Generic (PLEG): container finished" podID="d3340609-1895-4a4e-8dee-4319e1fecac0" containerID="944367e2e0c0b2ce3864374a20e2861a1899e24e428c50539bdb67be41adfe19" exitCode=0 Nov 02 20:22:25.338076 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:25.337675 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" event={"ID":"d3340609-1895-4a4e-8dee-4319e1fecac0","Type":"ContainerDied","Data":"944367e2e0c0b2ce3864374a20e2861a1899e24e428c50539bdb67be41adfe19"} Nov 02 20:22:27.182108 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.181764 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:27.304966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.304861 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3340609-1895-4a4e-8dee-4319e1fecac0-kube-api-access-z8kx8\" (UniqueName: \"kubernetes.io/projected/d3340609-1895-4a4e-8dee-4319e1fecac0-kube-api-access-z8kx8\") pod \"d3340609-1895-4a4e-8dee-4319e1fecac0\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " Nov 02 20:22:27.305679 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.305227 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-openshift-ca\") pod \"d3340609-1895-4a4e-8dee-4319e1fecac0\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " Nov 02 20:22:27.305679 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.305343 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-xks-ca\") pod \"d3340609-1895-4a4e-8dee-4319e1fecac0\" (UID: \"d3340609-1895-4a4e-8dee-4319e1fecac0\") " Nov 02 20:22:27.306232 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.306203 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-openshift-ca" pod "d3340609-1895-4a4e-8dee-4319e1fecac0" (UID: "d3340609-1895-4a4e-8dee-4319e1fecac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:22:27.322789 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.321219 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3340609-1895-4a4e-8dee-4319e1fecac0-kube-api-access-z8kx8" pod "d3340609-1895-4a4e-8dee-4319e1fecac0" (UID: "d3340609-1895-4a4e-8dee-4319e1fecac0"). InnerVolumeSpecName "kube-api-access-z8kx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:22:27.322789 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.321321 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-xks-ca" pod "d3340609-1895-4a4e-8dee-4319e1fecac0" (UID: "d3340609-1895-4a4e-8dee-4319e1fecac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:22:27.414075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.413862 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:27.414075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.413901 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8kx8\" (UniqueName: \"kubernetes.io/projected/d3340609-1895-4a4e-8dee-4319e1fecac0-kube-api-access-z8kx8\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:27.414075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.413919 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3340609-1895-4a4e-8dee-4319e1fecac0-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:27.676516 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.649908 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" event={"ID":"d3340609-1895-4a4e-8dee-4319e1fecac0","Type":"ContainerDied","Data":"f698c873d9ef07791e66ebbedbb7264529b6bd51fb8259edd1330f838b7eba1e"} Nov 02 20:22:27.676516 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.649976 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f698c873d9ef07791e66ebbedbb7264529b6bd51fb8259edd1330f838b7eba1e" Nov 02 20:22:27.676516 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.650052 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh" Nov 02 20:22:27.744517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.740606 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hszrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:22:27.744517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:27.740707 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podUID="ac5dc2a4-1609-4d43-b359-19adbb7aaeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:22:44.393678 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:44.393631 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-k56z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:22:44.394337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:44.394278 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-k56z5" podUID="1d8c479b-a392-4f75-9e43-c25fe3b73ada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:22:44.812524 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:22:44.812348 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:22:57.474449 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.474399 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6"] Nov 02 20:22:57.475123 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.475100 5850 cpu_manager.go:414] "containerMap: removing container" podUID="d3340609-1895-4a4e-8dee-4319e1fecac0" containerName="revoke-keys" Nov 02 20:22:57.475170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.475128 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3340609-1895-4a4e-8dee-4319e1fecac0" containerName="revoke-keys" Nov 02 20:22:57.475333 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.475319 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3340609-1895-4a4e-8dee-4319e1fecac0" containerName="revoke-keys" Nov 02 20:22:57.479007 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.478986 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.534491 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.534434 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6"] Nov 02 20:22:57.583024 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.582994 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:22:57.583201 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.583082 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:22:57.613620 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.613582 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbtnc\" (UniqueName: \"kubernetes.io/projected/d7978678-a282-4214-9240-bb0cd1e910b6-kube-api-access-sbtnc\") pod \"maas-api-revoke-sub-ae594979-qlgl6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.613794 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.613725 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-qlgl6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.613840 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.613813 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-qlgl6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.714790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.714748 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbtnc\" (UniqueName: \"kubernetes.io/projected/d7978678-a282-4214-9240-bb0cd1e910b6-kube-api-access-sbtnc\") pod \"maas-api-revoke-sub-ae594979-qlgl6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.714976 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.714906 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-qlgl6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.714976 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.714946 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-qlgl6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.715690 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.715669 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-qlgl6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.717316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.717285 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-qlgl6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.775621 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.775546 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbtnc\" (UniqueName: \"kubernetes.io/projected/d7978678-a282-4214-9240-bb0cd1e910b6-kube-api-access-sbtnc\") pod \"maas-api-revoke-sub-ae594979-qlgl6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:57.791531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:57.791500 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:22:58.162549 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:22:58.162505 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7978678_a282_4214_9240_bb0cd1e910b6.slice/crio-7656d3399ab53e8c3b64d99f219ff842f2c22adeee39fcf78382ef6896d1d147 WatchSource:0}: Error finding container 7656d3399ab53e8c3b64d99f219ff842f2c22adeee39fcf78382ef6896d1d147: Status 404 returned error can't find the container with id 7656d3399ab53e8c3b64d99f219ff842f2c22adeee39fcf78382ef6896d1d147 Nov 02 20:22:58.163737 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:58.163713 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6"] Nov 02 20:22:58.853648 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:58.853546 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" event={"ID":"d7978678-a282-4214-9240-bb0cd1e910b6","Type":"ContainerStarted","Data":"aa8ce83f4746a03e380f8b2dc9fb67147d35c9b9446fc23d31039de12708605f"} Nov 02 20:22:58.853648 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:58.853615 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" event={"ID":"d7978678-a282-4214-9240-bb0cd1e910b6","Type":"ContainerStarted","Data":"7656d3399ab53e8c3b64d99f219ff842f2c22adeee39fcf78382ef6896d1d147"} Nov 02 20:22:58.970857 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:58.970796 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" podStartSLOduration=1.9707800199999999 podStartE2EDuration="1.97078002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:58.968900352 +0000 UTC m=+3237.311289432" watchObservedRunningTime="2026-11-02 20:22:58.97078002 +0000 UTC m=+3237.313169099" Nov 02 20:22:59.860724 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:59.860686 5850 generic.go:356] "Generic (PLEG): container finished" podID="d7978678-a282-4214-9240-bb0cd1e910b6" containerID="aa8ce83f4746a03e380f8b2dc9fb67147d35c9b9446fc23d31039de12708605f" exitCode=0 Nov 02 20:22:59.861213 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:22:59.860737 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" event={"ID":"d7978678-a282-4214-9240-bb0cd1e910b6","Type":"ContainerDied","Data":"aa8ce83f4746a03e380f8b2dc9fb67147d35c9b9446fc23d31039de12708605f"} Nov 02 20:23:01.001440 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.001413 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:23:01.049746 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.049707 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-openshift-ca\") pod \"d7978678-a282-4214-9240-bb0cd1e910b6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " Nov 02 20:23:01.049985 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.049809 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7978678-a282-4214-9240-bb0cd1e910b6-kube-api-access-sbtnc\" (UniqueName: \"kubernetes.io/projected/d7978678-a282-4214-9240-bb0cd1e910b6-kube-api-access-sbtnc\") pod \"d7978678-a282-4214-9240-bb0cd1e910b6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " Nov 02 20:23:01.049985 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.049950 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-xks-ca\") pod \"d7978678-a282-4214-9240-bb0cd1e910b6\" (UID: \"d7978678-a282-4214-9240-bb0cd1e910b6\") " Nov 02 20:23:01.050175 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.050153 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-openshift-ca" pod "d7978678-a282-4214-9240-bb0cd1e910b6" (UID: "d7978678-a282-4214-9240-bb0cd1e910b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:01.050320 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.050298 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:01.051985 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.051952 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7978678-a282-4214-9240-bb0cd1e910b6-kube-api-access-sbtnc" pod "d7978678-a282-4214-9240-bb0cd1e910b6" (UID: "d7978678-a282-4214-9240-bb0cd1e910b6"). InnerVolumeSpecName "kube-api-access-sbtnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:01.051985 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.051973 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-xks-ca" pod "d7978678-a282-4214-9240-bb0cd1e910b6" (UID: "d7978678-a282-4214-9240-bb0cd1e910b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:01.151696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.151649 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbtnc\" (UniqueName: \"kubernetes.io/projected/d7978678-a282-4214-9240-bb0cd1e910b6-kube-api-access-sbtnc\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:01.151696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.151682 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7978678-a282-4214-9240-bb0cd1e910b6-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:01.871593 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.871562 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" Nov 02 20:23:01.871778 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.871597 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6" event={"ID":"d7978678-a282-4214-9240-bb0cd1e910b6","Type":"ContainerDied","Data":"7656d3399ab53e8c3b64d99f219ff842f2c22adeee39fcf78382ef6896d1d147"} Nov 02 20:23:01.871778 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:01.871645 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7656d3399ab53e8c3b64d99f219ff842f2c22adeee39fcf78382ef6896d1d147" Nov 02 20:23:14.761740 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:23:14.761652 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:23:21.498177 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:21.498136 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:23.048448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:23.048403 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg"] Nov 02 20:23:23.099842 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:23.099807 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cd7324-vdhbg"] Nov 02 20:23:23.813876 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:23.813837 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="580a7a33-0967-4f28-90b6-f9f3d117446b" path="/var/lib/kubelet/pods/580a7a33-0967-4f28-90b6-f9f3d117446b/volumes" Nov 02 20:23:25.354792 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.354755 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx"] Nov 02 20:23:25.355394 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.355372 5850 cpu_manager.go:414] "containerMap: removing container" podUID="d7978678-a282-4214-9240-bb0cd1e910b6" containerName="revoke-keys" Nov 02 20:23:25.355504 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.355397 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7978678-a282-4214-9240-bb0cd1e910b6" containerName="revoke-keys" Nov 02 20:23:25.355573 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.355527 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7978678-a282-4214-9240-bb0cd1e910b6" containerName="revoke-keys" Nov 02 20:23:25.358932 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.358910 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.414631 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.414595 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx"] Nov 02 20:23:25.464388 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.464357 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:23:25.464639 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.464428 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:23:25.473901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.473862 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vzkxx\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.474118 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.474000 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vzkxx\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.474210 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.474119 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frfnm\" (UniqueName: \"kubernetes.io/projected/1603608e-16f2-47b2-b3e0-ae60a4f1699c-kube-api-access-frfnm\") pod \"maas-api-revoke-sub-a4aef7d7-vzkxx\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.575352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.575313 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frfnm\" (UniqueName: \"kubernetes.io/projected/1603608e-16f2-47b2-b3e0-ae60a4f1699c-kube-api-access-frfnm\") pod \"maas-api-revoke-sub-a4aef7d7-vzkxx\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.575577 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.575368 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vzkxx\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.575577 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.575547 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vzkxx\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.576136 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.576117 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vzkxx\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.577963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.577942 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vzkxx\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.634163 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.634129 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frfnm\" (UniqueName: \"kubernetes.io/projected/1603608e-16f2-47b2-b3e0-ae60a4f1699c-kube-api-access-frfnm\") pod \"maas-api-revoke-sub-a4aef7d7-vzkxx\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:25.672127 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:25.672078 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:26.043598 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:23:26.043563 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1603608e_16f2_47b2_b3e0_ae60a4f1699c.slice/crio-8e4acd5d19333669656819636612f0ef946f7adbc55c8465972a60d19e7d0157 WatchSource:0}: Error finding container 8e4acd5d19333669656819636612f0ef946f7adbc55c8465972a60d19e7d0157: Status 404 returned error can't find the container with id 8e4acd5d19333669656819636612f0ef946f7adbc55c8465972a60d19e7d0157 Nov 02 20:23:26.045014 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:26.044990 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx"] Nov 02 20:23:27.031568 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:27.031534 5850 generic.go:356] "Generic (PLEG): container finished" podID="1603608e-16f2-47b2-b3e0-ae60a4f1699c" containerID="f7b693dfbb8fe36bc37bc9eac2f3e1fe0610a52a9ac9cd218b51e465158a0631" exitCode=0 Nov 02 20:23:27.032032 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:27.031596 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" event={"ID":"1603608e-16f2-47b2-b3e0-ae60a4f1699c","Type":"ContainerDied","Data":"f7b693dfbb8fe36bc37bc9eac2f3e1fe0610a52a9ac9cd218b51e465158a0631"} Nov 02 20:23:27.032032 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:27.031641 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" event={"ID":"1603608e-16f2-47b2-b3e0-ae60a4f1699c","Type":"ContainerStarted","Data":"8e4acd5d19333669656819636612f0ef946f7adbc55c8465972a60d19e7d0157"} Nov 02 20:23:28.175392 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.175364 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:28.204739 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.204703 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-openshift-ca\") pod \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " Nov 02 20:23:28.204917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.204799 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-xks-ca\") pod \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " Nov 02 20:23:28.204917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.204846 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1603608e-16f2-47b2-b3e0-ae60a4f1699c-kube-api-access-frfnm\" (UniqueName: \"kubernetes.io/projected/1603608e-16f2-47b2-b3e0-ae60a4f1699c-kube-api-access-frfnm\") pod \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\" (UID: \"1603608e-16f2-47b2-b3e0-ae60a4f1699c\") " Nov 02 20:23:28.205142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.205114 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-openshift-ca" pod "1603608e-16f2-47b2-b3e0-ae60a4f1699c" (UID: "1603608e-16f2-47b2-b3e0-ae60a4f1699c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:28.206951 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.206923 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-xks-ca" pod "1603608e-16f2-47b2-b3e0-ae60a4f1699c" (UID: "1603608e-16f2-47b2-b3e0-ae60a4f1699c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:28.207065 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.207021 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1603608e-16f2-47b2-b3e0-ae60a4f1699c-kube-api-access-frfnm" pod "1603608e-16f2-47b2-b3e0-ae60a4f1699c" (UID: "1603608e-16f2-47b2-b3e0-ae60a4f1699c"). InnerVolumeSpecName "kube-api-access-frfnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:28.306392 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.306300 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:28.306392 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.306331 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1603608e-16f2-47b2-b3e0-ae60a4f1699c-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:28.306392 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:28.306342 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frfnm\" (UniqueName: \"kubernetes.io/projected/1603608e-16f2-47b2-b3e0-ae60a4f1699c-kube-api-access-frfnm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:29.043113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:29.043079 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" Nov 02 20:23:29.043300 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:29.043109 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx" event={"ID":"1603608e-16f2-47b2-b3e0-ae60a4f1699c","Type":"ContainerDied","Data":"8e4acd5d19333669656819636612f0ef946f7adbc55c8465972a60d19e7d0157"} Nov 02 20:23:29.043300 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:29.043160 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e4acd5d19333669656819636612f0ef946f7adbc55c8465972a60d19e7d0157" Nov 02 20:23:42.046730 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:42.046693 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg"] Nov 02 20:23:42.101510 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:42.100656 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8062eb3d-sqnqg"] Nov 02 20:23:43.813764 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:43.813730 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341add44-2864-4b3c-bdf2-7bb1712569c2" path="/var/lib/kubelet/pods/341add44-2864-4b3c-bdf2-7bb1712569c2/volumes" Nov 02 20:23:44.761715 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:23:44.761678 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:23:55.895153 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:55.895108 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs"] Nov 02 20:23:55.895619 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:55.895569 5850 cpu_manager.go:414] "containerMap: removing container" podUID="1603608e-16f2-47b2-b3e0-ae60a4f1699c" containerName="revoke-keys" Nov 02 20:23:55.895619 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:55.895583 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1603608e-16f2-47b2-b3e0-ae60a4f1699c" containerName="revoke-keys" Nov 02 20:23:55.895696 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:55.895654 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="1603608e-16f2-47b2-b3e0-ae60a4f1699c" containerName="revoke-keys" Nov 02 20:23:55.898729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:55.898712 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:55.925994 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:55.925958 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xcbxs\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:55.926187 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:55.926018 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xcbxs\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:55.926187 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:55.926110 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqm8g\" (UniqueName: \"kubernetes.io/projected/fe6080b2-d5c7-472d-b895-45a98d83ecc5-kube-api-access-rqm8g\") pod \"maas-api-revoke-sub-79805ca9-xcbxs\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:56.004936 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.004894 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:23:56.005117 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.004942 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:23:56.006964 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.006945 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs"] Nov 02 20:23:56.026771 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.026738 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xcbxs\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:56.026889 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.026794 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xcbxs\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:56.026889 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.026817 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqm8g\" (UniqueName: \"kubernetes.io/projected/fe6080b2-d5c7-472d-b895-45a98d83ecc5-kube-api-access-rqm8g\") pod \"maas-api-revoke-sub-79805ca9-xcbxs\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:56.027465 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.027441 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xcbxs\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:56.029246 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.029221 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xcbxs\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:56.085869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.085829 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqm8g\" (UniqueName: \"kubernetes.io/projected/fe6080b2-d5c7-472d-b895-45a98d83ecc5-kube-api-access-rqm8g\") pod \"maas-api-revoke-sub-79805ca9-xcbxs\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:56.211838 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.211749 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:56.581105 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:23:56.581068 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6080b2_d5c7_472d_b895_45a98d83ecc5.slice/crio-f7eaa81d807358b0f2ce90f9d5b0baa68ed88699f3dfd6f54f457f1a00a6a86a WatchSource:0}: Error finding container f7eaa81d807358b0f2ce90f9d5b0baa68ed88699f3dfd6f54f457f1a00a6a86a: Status 404 returned error can't find the container with id f7eaa81d807358b0f2ce90f9d5b0baa68ed88699f3dfd6f54f457f1a00a6a86a Nov 02 20:23:56.582552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:56.582522 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs"] Nov 02 20:23:57.228658 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:57.228566 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" event={"ID":"fe6080b2-d5c7-472d-b895-45a98d83ecc5","Type":"ContainerStarted","Data":"698358e2a8c95afea6a98d00be6dfdb1055a5e926aeec27c820ea17bfa1bb55b"} Nov 02 20:23:57.228658 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:57.228624 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" event={"ID":"fe6080b2-d5c7-472d-b895-45a98d83ecc5","Type":"ContainerStarted","Data":"f7eaa81d807358b0f2ce90f9d5b0baa68ed88699f3dfd6f54f457f1a00a6a86a"} Nov 02 20:23:57.345986 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:57.345925 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" podStartSLOduration=2.345909685 podStartE2EDuration="2.345909685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:57.34399231 +0000 UTC m=+3295.686381391" watchObservedRunningTime="2026-11-02 20:23:57.345909685 +0000 UTC m=+3295.688298762" Nov 02 20:23:58.235558 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:58.235523 5850 generic.go:356] "Generic (PLEG): container finished" podID="fe6080b2-d5c7-472d-b895-45a98d83ecc5" containerID="698358e2a8c95afea6a98d00be6dfdb1055a5e926aeec27c820ea17bfa1bb55b" exitCode=0 Nov 02 20:23:58.235954 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:58.235584 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" event={"ID":"fe6080b2-d5c7-472d-b895-45a98d83ecc5","Type":"ContainerDied","Data":"698358e2a8c95afea6a98d00be6dfdb1055a5e926aeec27c820ea17bfa1bb55b"} Nov 02 20:23:59.377757 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.377730 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:23:59.462753 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.462711 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-openshift-ca\") pod \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " Nov 02 20:23:59.462934 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.462766 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6080b2-d5c7-472d-b895-45a98d83ecc5-kube-api-access-rqm8g\" (UniqueName: \"kubernetes.io/projected/fe6080b2-d5c7-472d-b895-45a98d83ecc5-kube-api-access-rqm8g\") pod \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " Nov 02 20:23:59.462934 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.462867 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-xks-ca\") pod \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\" (UID: \"fe6080b2-d5c7-472d-b895-45a98d83ecc5\") " Nov 02 20:23:59.463110 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.463084 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-openshift-ca" pod "fe6080b2-d5c7-472d-b895-45a98d83ecc5" (UID: "fe6080b2-d5c7-472d-b895-45a98d83ecc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:59.463265 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.463240 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:59.464987 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.464960 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-xks-ca" pod "fe6080b2-d5c7-472d-b895-45a98d83ecc5" (UID: "fe6080b2-d5c7-472d-b895-45a98d83ecc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:59.465070 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.465015 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6080b2-d5c7-472d-b895-45a98d83ecc5-kube-api-access-rqm8g" pod "fe6080b2-d5c7-472d-b895-45a98d83ecc5" (UID: "fe6080b2-d5c7-472d-b895-45a98d83ecc5"). InnerVolumeSpecName "kube-api-access-rqm8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:59.564540 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.564420 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqm8g\" (UniqueName: \"kubernetes.io/projected/fe6080b2-d5c7-472d-b895-45a98d83ecc5-kube-api-access-rqm8g\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:59.564540 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:23:59.564456 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6080b2-d5c7-472d-b895-45a98d83ecc5-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:00.247167 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:00.247131 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" event={"ID":"fe6080b2-d5c7-472d-b895-45a98d83ecc5","Type":"ContainerDied","Data":"f7eaa81d807358b0f2ce90f9d5b0baa68ed88699f3dfd6f54f457f1a00a6a86a"} Nov 02 20:24:00.247360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:00.247174 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs" Nov 02 20:24:00.247360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:00.247180 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7eaa81d807358b0f2ce90f9d5b0baa68ed88699f3dfd6f54f457f1a00a6a86a" Nov 02 20:24:12.728642 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:12.728591 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hszrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:24:12.729349 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:12.729303 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hszrp" podUID="ac5dc2a4-1609-4d43-b359-19adbb7aaeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:24:13.077701 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:13.077587 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d2chl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:24:13.078166 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:13.078128 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d2chl" podUID="6222ddb5-cd09-43db-82f2-01c85b1e63d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:24:14.759965 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:24:14.759925 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:24:16.467173 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.467130 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m"] Nov 02 20:24:16.467603 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.467587 5850 cpu_manager.go:414] "containerMap: removing container" podUID="fe6080b2-d5c7-472d-b895-45a98d83ecc5" containerName="revoke-keys" Nov 02 20:24:16.467655 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.467604 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6080b2-d5c7-472d-b895-45a98d83ecc5" containerName="revoke-keys" Nov 02 20:24:16.467713 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.467702 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6080b2-d5c7-472d-b895-45a98d83ecc5" containerName="revoke-keys" Nov 02 20:24:16.471718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.471692 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.525979 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.525943 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:24:16.529049 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.529017 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m"] Nov 02 20:24:16.577168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.577132 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:24:16.604198 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.604159 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57380394-db7f-482e-9ce5-093323f97663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6q52m\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.604410 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.604224 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57380394-db7f-482e-9ce5-093323f97663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6q52m\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.604410 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.604313 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmf46\" (UniqueName: \"kubernetes.io/projected/57380394-db7f-482e-9ce5-093323f97663-kube-api-access-qmf46\") pod \"maas-api-revoke-sub-0b7e8edb-6q52m\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.705153 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.705109 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57380394-db7f-482e-9ce5-093323f97663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6q52m\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.705370 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.705173 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57380394-db7f-482e-9ce5-093323f97663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6q52m\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.705370 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.705199 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmf46\" (UniqueName: \"kubernetes.io/projected/57380394-db7f-482e-9ce5-093323f97663-kube-api-access-qmf46\") pod \"maas-api-revoke-sub-0b7e8edb-6q52m\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.705922 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.705895 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57380394-db7f-482e-9ce5-093323f97663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6q52m\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.707688 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.707670 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57380394-db7f-482e-9ce5-093323f97663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6q52m\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.763959 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.763883 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmf46\" (UniqueName: \"kubernetes.io/projected/57380394-db7f-482e-9ce5-093323f97663-kube-api-access-qmf46\") pod \"maas-api-revoke-sub-0b7e8edb-6q52m\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:16.789668 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:16.789628 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:17.158463 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:24:17.158431 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57380394_db7f_482e_9ce5_093323f97663.slice/crio-f022061ac945f00b4756c4198c88c5f952f85ea294425588c86411c41e8fb261 WatchSource:0}: Error finding container f022061ac945f00b4756c4198c88c5f952f85ea294425588c86411c41e8fb261: Status 404 returned error can't find the container with id f022061ac945f00b4756c4198c88c5f952f85ea294425588c86411c41e8fb261 Nov 02 20:24:17.159902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:17.159862 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m"] Nov 02 20:24:17.354361 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:17.354328 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" event={"ID":"57380394-db7f-482e-9ce5-093323f97663","Type":"ContainerStarted","Data":"f022061ac945f00b4756c4198c88c5f952f85ea294425588c86411c41e8fb261"} Nov 02 20:24:18.361286 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:18.361197 5850 generic.go:356] "Generic (PLEG): container finished" podID="57380394-db7f-482e-9ce5-093323f97663" containerID="9d4f5b566db36dd8ee17861eca7d219c151ff75ee7fc1c5550fc31ff223d6c37" exitCode=0 Nov 02 20:24:18.361286 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:18.361272 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" event={"ID":"57380394-db7f-482e-9ce5-093323f97663","Type":"ContainerDied","Data":"9d4f5b566db36dd8ee17861eca7d219c151ff75ee7fc1c5550fc31ff223d6c37"} Nov 02 20:24:19.501346 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.501313 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:19.634022 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.633988 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57380394-db7f-482e-9ce5-093323f97663-kube-api-access-qmf46\" (UniqueName: \"kubernetes.io/projected/57380394-db7f-482e-9ce5-093323f97663-kube-api-access-qmf46\") pod \"57380394-db7f-482e-9ce5-093323f97663\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " Nov 02 20:24:19.634214 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.634032 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57380394-db7f-482e-9ce5-093323f97663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57380394-db7f-482e-9ce5-093323f97663-maas-api-xks-ca\") pod \"57380394-db7f-482e-9ce5-093323f97663\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " Nov 02 20:24:19.634214 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.634155 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57380394-db7f-482e-9ce5-093323f97663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57380394-db7f-482e-9ce5-093323f97663-maas-api-openshift-ca\") pod \"57380394-db7f-482e-9ce5-093323f97663\" (UID: \"57380394-db7f-482e-9ce5-093323f97663\") " Nov 02 20:24:19.634635 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.634605 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57380394-db7f-482e-9ce5-093323f97663-maas-api-openshift-ca" pod "57380394-db7f-482e-9ce5-093323f97663" (UID: "57380394-db7f-482e-9ce5-093323f97663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:19.636303 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.636274 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57380394-db7f-482e-9ce5-093323f97663-maas-api-xks-ca" pod "57380394-db7f-482e-9ce5-093323f97663" (UID: "57380394-db7f-482e-9ce5-093323f97663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:24:19.636401 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.636278 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57380394-db7f-482e-9ce5-093323f97663-kube-api-access-qmf46" pod "57380394-db7f-482e-9ce5-093323f97663" (UID: "57380394-db7f-482e-9ce5-093323f97663"). InnerVolumeSpecName "kube-api-access-qmf46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:19.735507 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.735435 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57380394-db7f-482e-9ce5-093323f97663-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:19.735507 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.735499 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmf46\" (UniqueName: \"kubernetes.io/projected/57380394-db7f-482e-9ce5-093323f97663-kube-api-access-qmf46\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:19.735507 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:19.735510 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57380394-db7f-482e-9ce5-093323f97663-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:20.372559 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:20.372529 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" Nov 02 20:24:20.372728 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:20.372523 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m" event={"ID":"57380394-db7f-482e-9ce5-093323f97663","Type":"ContainerDied","Data":"f022061ac945f00b4756c4198c88c5f952f85ea294425588c86411c41e8fb261"} Nov 02 20:24:20.372728 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:20.372656 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f022061ac945f00b4756c4198c88c5f952f85ea294425588c86411c41e8fb261" Nov 02 20:24:21.874683 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:21.874648 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:23.519378 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:23.519294 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:27.771123 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.771086 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft"] Nov 02 20:24:27.771710 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.771618 5850 cpu_manager.go:414] "containerMap: removing container" podUID="57380394-db7f-482e-9ce5-093323f97663" containerName="revoke-keys" Nov 02 20:24:27.771710 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.771633 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57380394-db7f-482e-9ce5-093323f97663" containerName="revoke-keys" Nov 02 20:24:27.771833 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.771728 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="57380394-db7f-482e-9ce5-093323f97663" containerName="revoke-keys" Nov 02 20:24:27.775097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.775076 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:27.814009 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.813967 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sdtft\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:27.814235 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.814040 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5ns\" (UniqueName: \"kubernetes.io/projected/835afa3e-c554-4462-95b5-2cc2814e2768-kube-api-access-7v5ns\") pod \"maas-api-revoke-sub-3d4156f1-sdtft\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:27.814235 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.814137 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sdtft\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:27.832402 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.832365 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft"] Nov 02 20:24:27.881260 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.881222 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:24:27.881451 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.881303 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:24:27.915450 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.915411 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5ns\" (UniqueName: \"kubernetes.io/projected/835afa3e-c554-4462-95b5-2cc2814e2768-kube-api-access-7v5ns\") pod \"maas-api-revoke-sub-3d4156f1-sdtft\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:27.915450 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.915449 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sdtft\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:27.915751 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.915548 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sdtft\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:27.916192 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.916164 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sdtft\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:27.918012 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.917983 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sdtft\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:27.974385 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:27.974352 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5ns\" (UniqueName: \"kubernetes.io/projected/835afa3e-c554-4462-95b5-2cc2814e2768-kube-api-access-7v5ns\") pod \"maas-api-revoke-sub-3d4156f1-sdtft\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:28.088831 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:28.088710 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:28.456562 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:24:28.456530 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835afa3e_c554_4462_95b5_2cc2814e2768.slice/crio-4d5ae63b5ff839bc2c8fb3ab912f43464bd1ed5776b5f2798acd11fb01f88d3a WatchSource:0}: Error finding container 4d5ae63b5ff839bc2c8fb3ab912f43464bd1ed5776b5f2798acd11fb01f88d3a: Status 404 returned error can't find the container with id 4d5ae63b5ff839bc2c8fb3ab912f43464bd1ed5776b5f2798acd11fb01f88d3a Nov 02 20:24:28.458953 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:28.458929 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft"] Nov 02 20:24:29.421165 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:29.421131 5850 generic.go:356] "Generic (PLEG): container finished" podID="835afa3e-c554-4462-95b5-2cc2814e2768" containerID="09862f03bfed52ee017c45c5b940c3cc4b8457911bddb4a1afe05e4f442c1b72" exitCode=0 Nov 02 20:24:29.421626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:29.421204 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" event={"ID":"835afa3e-c554-4462-95b5-2cc2814e2768","Type":"ContainerDied","Data":"09862f03bfed52ee017c45c5b940c3cc4b8457911bddb4a1afe05e4f442c1b72"} Nov 02 20:24:29.421626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:29.421250 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" event={"ID":"835afa3e-c554-4462-95b5-2cc2814e2768","Type":"ContainerStarted","Data":"4d5ae63b5ff839bc2c8fb3ab912f43464bd1ed5776b5f2798acd11fb01f88d3a"} Nov 02 20:24:30.560177 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.560150 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:30.642175 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.642135 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-xks-ca\") pod \"835afa3e-c554-4462-95b5-2cc2814e2768\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " Nov 02 20:24:30.642348 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.642236 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-openshift-ca\") pod \"835afa3e-c554-4462-95b5-2cc2814e2768\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " Nov 02 20:24:30.642348 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.642307 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835afa3e-c554-4462-95b5-2cc2814e2768-kube-api-access-7v5ns\" (UniqueName: \"kubernetes.io/projected/835afa3e-c554-4462-95b5-2cc2814e2768-kube-api-access-7v5ns\") pod \"835afa3e-c554-4462-95b5-2cc2814e2768\" (UID: \"835afa3e-c554-4462-95b5-2cc2814e2768\") " Nov 02 20:24:30.642664 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.642638 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-openshift-ca" pod "835afa3e-c554-4462-95b5-2cc2814e2768" (UID: "835afa3e-c554-4462-95b5-2cc2814e2768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:30.644388 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.644364 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-xks-ca" pod "835afa3e-c554-4462-95b5-2cc2814e2768" (UID: "835afa3e-c554-4462-95b5-2cc2814e2768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:24:30.644497 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.644427 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835afa3e-c554-4462-95b5-2cc2814e2768-kube-api-access-7v5ns" pod "835afa3e-c554-4462-95b5-2cc2814e2768" (UID: "835afa3e-c554-4462-95b5-2cc2814e2768"). InnerVolumeSpecName "kube-api-access-7v5ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:30.743388 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.743283 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:30.743388 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.743322 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v5ns\" (UniqueName: \"kubernetes.io/projected/835afa3e-c554-4462-95b5-2cc2814e2768-kube-api-access-7v5ns\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:30.743388 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:30.743337 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835afa3e-c554-4462-95b5-2cc2814e2768-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:31.431850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:31.431816 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" Nov 02 20:24:31.431850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:31.431826 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft" event={"ID":"835afa3e-c554-4462-95b5-2cc2814e2768","Type":"ContainerDied","Data":"4d5ae63b5ff839bc2c8fb3ab912f43464bd1ed5776b5f2798acd11fb01f88d3a"} Nov 02 20:24:31.432128 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:31.431872 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d5ae63b5ff839bc2c8fb3ab912f43464bd1ed5776b5f2798acd11fb01f88d3a" Nov 02 20:24:35.055190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:35.055146 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f"] Nov 02 20:24:35.106756 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:35.106713 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bjr5f"] Nov 02 20:24:35.814330 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:35.814296 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178183e7-e489-4178-99c3-5309122770e1" path="/var/lib/kubelet/pods/178183e7-e489-4178-99c3-5309122770e1/volumes" Nov 02 20:24:44.897895 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:24:44.897670 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:24:48.912082 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:48.912047 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft"] Nov 02 20:24:48.963554 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:48.963518 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sdtft"] Nov 02 20:24:49.068918 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.068881 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4"] Nov 02 20:24:49.069346 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.069333 5850 cpu_manager.go:414] "containerMap: removing container" podUID="835afa3e-c554-4462-95b5-2cc2814e2768" containerName="revoke-keys" Nov 02 20:24:49.069414 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.069347 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835afa3e-c554-4462-95b5-2cc2814e2768" containerName="revoke-keys" Nov 02 20:24:49.069457 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.069428 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="835afa3e-c554-4462-95b5-2cc2814e2768" containerName="revoke-keys" Nov 02 20:24:49.074013 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.073989 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.128009 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.127975 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:24:49.128462 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.128435 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:24:49.128661 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.128640 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4"] Nov 02 20:24:49.174643 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.174550 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gksh4\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.174810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.174697 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbddd\" (UniqueName: \"kubernetes.io/projected/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-kube-api-access-dbddd\") pod \"maas-api-revoke-sub-3d4156f1-gksh4\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.174810 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.174763 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gksh4\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.275749 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.275706 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbddd\" (UniqueName: \"kubernetes.io/projected/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-kube-api-access-dbddd\") pod \"maas-api-revoke-sub-3d4156f1-gksh4\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.275947 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.275770 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gksh4\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.275947 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.275807 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gksh4\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.276485 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.276444 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gksh4\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.278292 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.278264 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gksh4\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.334411 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.334372 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbddd\" (UniqueName: \"kubernetes.io/projected/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-kube-api-access-dbddd\") pod \"maas-api-revoke-sub-3d4156f1-gksh4\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.386859 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.386816 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:49.754622 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:24:49.754586 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942bb383_1bd6_4bc5_8ca4_73f9382e49ed.slice/crio-a700ef330d7faa0cfe808c7b6b9fd0f740a423872654bd84ecef9f9074118520 WatchSource:0}: Error finding container a700ef330d7faa0cfe808c7b6b9fd0f740a423872654bd84ecef9f9074118520: Status 404 returned error can't find the container with id a700ef330d7faa0cfe808c7b6b9fd0f740a423872654bd84ecef9f9074118520 Nov 02 20:24:49.756632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.756606 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4"] Nov 02 20:24:49.813570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:49.813540 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835afa3e-c554-4462-95b5-2cc2814e2768" path="/var/lib/kubelet/pods/835afa3e-c554-4462-95b5-2cc2814e2768/volumes" Nov 02 20:24:50.560898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:50.560860 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" event={"ID":"942bb383-1bd6-4bc5-8ca4-73f9382e49ed","Type":"ContainerStarted","Data":"8add7afd38d2c349a887ac7b257bdfcb27b65a64f6b2ffd573bdb46f402d326a"} Nov 02 20:24:50.561419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:50.560914 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" event={"ID":"942bb383-1bd6-4bc5-8ca4-73f9382e49ed","Type":"ContainerStarted","Data":"a700ef330d7faa0cfe808c7b6b9fd0f740a423872654bd84ecef9f9074118520"} Nov 02 20:24:50.676686 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:50.676631 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" podStartSLOduration=1.676618333 podStartE2EDuration="1.676618333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:50.676565575 +0000 UTC m=+3349.018954656" watchObservedRunningTime="2026-11-02 20:24:50.676618333 +0000 UTC m=+3349.019007412" Nov 02 20:24:51.567575 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:51.567539 5850 generic.go:356] "Generic (PLEG): container finished" podID="942bb383-1bd6-4bc5-8ca4-73f9382e49ed" containerID="8add7afd38d2c349a887ac7b257bdfcb27b65a64f6b2ffd573bdb46f402d326a" exitCode=0 Nov 02 20:24:51.567977 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:51.567589 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" event={"ID":"942bb383-1bd6-4bc5-8ca4-73f9382e49ed","Type":"ContainerDied","Data":"8add7afd38d2c349a887ac7b257bdfcb27b65a64f6b2ffd573bdb46f402d326a"} Nov 02 20:24:52.723400 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:52.723371 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:52.912367 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:52.912332 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-xks-ca\") pod \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " Nov 02 20:24:52.912585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:52.912394 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-kube-api-access-dbddd\" (UniqueName: \"kubernetes.io/projected/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-kube-api-access-dbddd\") pod \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " Nov 02 20:24:52.912585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:52.912491 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-openshift-ca\") pod \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\" (UID: \"942bb383-1bd6-4bc5-8ca4-73f9382e49ed\") " Nov 02 20:24:52.913557 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:52.912877 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-openshift-ca" pod "942bb383-1bd6-4bc5-8ca4-73f9382e49ed" (UID: "942bb383-1bd6-4bc5-8ca4-73f9382e49ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:52.913557 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:52.913178 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:52.914870 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:52.914846 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-xks-ca" pod "942bb383-1bd6-4bc5-8ca4-73f9382e49ed" (UID: "942bb383-1bd6-4bc5-8ca4-73f9382e49ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:24:52.914964 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:52.914886 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-kube-api-access-dbddd" pod "942bb383-1bd6-4bc5-8ca4-73f9382e49ed" (UID: "942bb383-1bd6-4bc5-8ca4-73f9382e49ed"). InnerVolumeSpecName "kube-api-access-dbddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:53.014409 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:53.014373 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:53.014409 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:53.014403 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbddd\" (UniqueName: \"kubernetes.io/projected/942bb383-1bd6-4bc5-8ca4-73f9382e49ed-kube-api-access-dbddd\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:53.586533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:53.586496 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" Nov 02 20:24:53.586700 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:53.586506 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4" event={"ID":"942bb383-1bd6-4bc5-8ca4-73f9382e49ed","Type":"ContainerDied","Data":"a700ef330d7faa0cfe808c7b6b9fd0f740a423872654bd84ecef9f9074118520"} Nov 02 20:24:53.586700 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:24:53.586644 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a700ef330d7faa0cfe808c7b6b9fd0f740a423872654bd84ecef9f9074118520" Nov 02 20:25:00.170170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:00.170118 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:03.668084 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:03.668049 5850 scope.go:122] "RemoveContainer" containerID="069d5a7db2485f05e26a2fe39e12e18b9cfb68cc3bf88971519b2cf48e85ff84" Nov 02 20:25:03.679074 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:03.679051 5850 scope.go:122] "RemoveContainer" containerID="21e0ceef1a453c2940169f1b4ea603dd4657e6fc15f715848d23c67db65219a2" Nov 02 20:25:14.761890 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:25:14.761846 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:25:22.125905 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:22.125868 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:22.498825 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:22.498745 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:23.769172 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.769135 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm"] Nov 02 20:25:23.769784 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.769762 5850 cpu_manager.go:414] "containerMap: removing container" podUID="942bb383-1bd6-4bc5-8ca4-73f9382e49ed" containerName="revoke-keys" Nov 02 20:25:23.769885 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.769786 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942bb383-1bd6-4bc5-8ca4-73f9382e49ed" containerName="revoke-keys" Nov 02 20:25:23.769945 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.769929 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="942bb383-1bd6-4bc5-8ca4-73f9382e49ed" containerName="revoke-keys" Nov 02 20:25:23.774932 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.774905 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:23.831035 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.830999 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm"] Nov 02 20:25:23.868050 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.868010 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36489392-98d5-4216-a25a-169ff5958406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gffcm\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:23.868282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.868132 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96lwp\" (UniqueName: \"kubernetes.io/projected/36489392-98d5-4216-a25a-169ff5958406-kube-api-access-96lwp\") pod \"maas-api-revoke-sub-e2fc9c5b-gffcm\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:23.868282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.868218 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36489392-98d5-4216-a25a-169ff5958406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gffcm\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:23.879813 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.879784 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:25:23.879946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.879887 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:25:23.969363 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.969324 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36489392-98d5-4216-a25a-169ff5958406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gffcm\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:23.969605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.969383 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36489392-98d5-4216-a25a-169ff5958406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gffcm\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:23.969605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.969541 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96lwp\" (UniqueName: \"kubernetes.io/projected/36489392-98d5-4216-a25a-169ff5958406-kube-api-access-96lwp\") pod \"maas-api-revoke-sub-e2fc9c5b-gffcm\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:23.970164 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.970135 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36489392-98d5-4216-a25a-169ff5958406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gffcm\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:23.971917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:23.971891 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36489392-98d5-4216-a25a-169ff5958406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gffcm\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:24.028797 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:24.028724 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96lwp\" (UniqueName: \"kubernetes.io/projected/36489392-98d5-4216-a25a-169ff5958406-kube-api-access-96lwp\") pod \"maas-api-revoke-sub-e2fc9c5b-gffcm\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:24.087791 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:24.087749 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:24.457544 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:25:24.457514 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36489392_98d5_4216_a25a_169ff5958406.slice/crio-803d6a774eca3e8c0c7b05beed2472728cfb5fe7c141c101f1d625dd452ae8cd WatchSource:0}: Error finding container 803d6a774eca3e8c0c7b05beed2472728cfb5fe7c141c101f1d625dd452ae8cd: Status 404 returned error can't find the container with id 803d6a774eca3e8c0c7b05beed2472728cfb5fe7c141c101f1d625dd452ae8cd Nov 02 20:25:24.458976 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:24.458951 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm"] Nov 02 20:25:24.765318 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:24.765278 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" event={"ID":"36489392-98d5-4216-a25a-169ff5958406","Type":"ContainerStarted","Data":"9512ed6f3b98ea18a6998dd04f43b3f011e6e74e51165c5cbff916ea28d87dfc"} Nov 02 20:25:24.765565 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:24.765328 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" event={"ID":"36489392-98d5-4216-a25a-169ff5958406","Type":"ContainerStarted","Data":"803d6a774eca3e8c0c7b05beed2472728cfb5fe7c141c101f1d625dd452ae8cd"} Nov 02 20:25:24.883162 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:24.883108 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" podStartSLOduration=1.883094627 podStartE2EDuration="1.883094627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:24.881332396 +0000 UTC m=+3383.223721477" watchObservedRunningTime="2026-11-02 20:25:24.883094627 +0000 UTC m=+3383.225483706" Nov 02 20:25:25.771396 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:25.771360 5850 generic.go:356] "Generic (PLEG): container finished" podID="36489392-98d5-4216-a25a-169ff5958406" containerID="9512ed6f3b98ea18a6998dd04f43b3f011e6e74e51165c5cbff916ea28d87dfc" exitCode=0 Nov 02 20:25:25.771605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:25.771457 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" event={"ID":"36489392-98d5-4216-a25a-169ff5958406","Type":"ContainerDied","Data":"9512ed6f3b98ea18a6998dd04f43b3f011e6e74e51165c5cbff916ea28d87dfc"} Nov 02 20:25:26.914595 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:26.914564 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:27.001547 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.001510 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36489392-98d5-4216-a25a-169ff5958406-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36489392-98d5-4216-a25a-169ff5958406-maas-api-openshift-ca\") pod \"36489392-98d5-4216-a25a-169ff5958406\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " Nov 02 20:25:27.001736 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.001558 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36489392-98d5-4216-a25a-169ff5958406-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36489392-98d5-4216-a25a-169ff5958406-maas-api-xks-ca\") pod \"36489392-98d5-4216-a25a-169ff5958406\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " Nov 02 20:25:27.001736 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.001601 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36489392-98d5-4216-a25a-169ff5958406-kube-api-access-96lwp\" (UniqueName: \"kubernetes.io/projected/36489392-98d5-4216-a25a-169ff5958406-kube-api-access-96lwp\") pod \"36489392-98d5-4216-a25a-169ff5958406\" (UID: \"36489392-98d5-4216-a25a-169ff5958406\") " Nov 02 20:25:27.001902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.001878 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36489392-98d5-4216-a25a-169ff5958406-maas-api-openshift-ca" pod "36489392-98d5-4216-a25a-169ff5958406" (UID: "36489392-98d5-4216-a25a-169ff5958406"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:27.003869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.003841 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36489392-98d5-4216-a25a-169ff5958406-kube-api-access-96lwp" pod "36489392-98d5-4216-a25a-169ff5958406" (UID: "36489392-98d5-4216-a25a-169ff5958406"). InnerVolumeSpecName "kube-api-access-96lwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:27.003869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.003839 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36489392-98d5-4216-a25a-169ff5958406-maas-api-xks-ca" pod "36489392-98d5-4216-a25a-169ff5958406" (UID: "36489392-98d5-4216-a25a-169ff5958406"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:25:27.102957 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.102871 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36489392-98d5-4216-a25a-169ff5958406-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:27.102957 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.102902 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36489392-98d5-4216-a25a-169ff5958406-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:27.102957 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.102915 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96lwp\" (UniqueName: \"kubernetes.io/projected/36489392-98d5-4216-a25a-169ff5958406-kube-api-access-96lwp\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:27.783330 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.783295 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" event={"ID":"36489392-98d5-4216-a25a-169ff5958406","Type":"ContainerDied","Data":"803d6a774eca3e8c0c7b05beed2472728cfb5fe7c141c101f1d625dd452ae8cd"} Nov 02 20:25:27.783533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.783347 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="803d6a774eca3e8c0c7b05beed2472728cfb5fe7c141c101f1d625dd452ae8cd" Nov 02 20:25:27.783533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.783373 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm" Nov 02 20:25:27.886347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:27.886295 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:28.552634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:28.552598 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:34.468864 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.468819 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5"] Nov 02 20:25:34.469655 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.469630 5850 cpu_manager.go:414] "containerMap: removing container" podUID="36489392-98d5-4216-a25a-169ff5958406" containerName="revoke-keys" Nov 02 20:25:34.469721 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.469660 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36489392-98d5-4216-a25a-169ff5958406" containerName="revoke-keys" Nov 02 20:25:34.469842 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.469826 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="36489392-98d5-4216-a25a-169ff5958406" containerName="revoke-keys" Nov 02 20:25:34.473681 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.473660 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.528968 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.528936 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5"] Nov 02 20:25:34.577756 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.577720 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:25:34.577963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.577881 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:25:34.582301 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.582275 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rskm5\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.582355 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.582341 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rskm5\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.582511 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.582459 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2h9\" (UniqueName: \"kubernetes.io/projected/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-kube-api-access-cb2h9\") pod \"maas-api-revoke-sub-5a29efb3-rskm5\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.683949 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.683905 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rskm5\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.684140 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.683974 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rskm5\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.684140 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.684012 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2h9\" (UniqueName: \"kubernetes.io/projected/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-kube-api-access-cb2h9\") pod \"maas-api-revoke-sub-5a29efb3-rskm5\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.684613 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.684591 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rskm5\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.686417 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.686391 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rskm5\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.742784 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.742702 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2h9\" (UniqueName: \"kubernetes.io/projected/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-kube-api-access-cb2h9\") pod \"maas-api-revoke-sub-5a29efb3-rskm5\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:34.789563 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:34.789520 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:35.159341 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:25:35.159303 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fa2f8e_6acb_47a2_bfd7_844eb3bc5f9f.slice/crio-ff32b88bd49ab785dfe574e2a62b5b4f8601b466cb7cdeacfd73502f06f12b70 WatchSource:0}: Error finding container ff32b88bd49ab785dfe574e2a62b5b4f8601b466cb7cdeacfd73502f06f12b70: Status 404 returned error can't find the container with id ff32b88bd49ab785dfe574e2a62b5b4f8601b466cb7cdeacfd73502f06f12b70 Nov 02 20:25:35.161709 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:35.161682 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5"] Nov 02 20:25:35.828426 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:35.828326 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" event={"ID":"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f","Type":"ContainerStarted","Data":"bb69ea514cf7dd411e2d437fe059f788309b3dd6b094d63378526a46e601c621"} Nov 02 20:25:35.828426 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:35.828382 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" event={"ID":"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f","Type":"ContainerStarted","Data":"ff32b88bd49ab785dfe574e2a62b5b4f8601b466cb7cdeacfd73502f06f12b70"} Nov 02 20:25:35.945436 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:35.945376 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" podStartSLOduration=1.94536093 podStartE2EDuration="1.94536093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:35.943830521 +0000 UTC m=+3394.286219615" watchObservedRunningTime="2026-11-02 20:25:35.94536093 +0000 UTC m=+3394.287750010" Nov 02 20:25:36.835570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:36.835536 5850 generic.go:356] "Generic (PLEG): container finished" podID="f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f" containerID="bb69ea514cf7dd411e2d437fe059f788309b3dd6b094d63378526a46e601c621" exitCode=0 Nov 02 20:25:36.836093 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:36.835660 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" event={"ID":"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f","Type":"ContainerDied","Data":"bb69ea514cf7dd411e2d437fe059f788309b3dd6b094d63378526a46e601c621"} Nov 02 20:25:38.314995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.314919 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:38.370506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.369809 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-openshift-ca\") pod \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " Nov 02 20:25:38.370506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.369871 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-kube-api-access-cb2h9\" (UniqueName: \"kubernetes.io/projected/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-kube-api-access-cb2h9\") pod \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " Nov 02 20:25:38.370506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.370000 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-xks-ca\") pod \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\" (UID: \"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f\") " Nov 02 20:25:38.378833 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.378768 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-openshift-ca" pod "f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f" (UID: "f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:38.395705 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.395630 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-xks-ca" pod "f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f" (UID: "f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:25:38.399155 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.399096 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-kube-api-access-cb2h9" pod "f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f" (UID: "f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f"). InnerVolumeSpecName "kube-api-access-cb2h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:38.471359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.471212 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:38.471359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.471251 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2h9\" (UniqueName: \"kubernetes.io/projected/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-kube-api-access-cb2h9\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:38.471359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.471270 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:38.882738 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.882682 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" event={"ID":"f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f","Type":"ContainerDied","Data":"ff32b88bd49ab785dfe574e2a62b5b4f8601b466cb7cdeacfd73502f06f12b70"} Nov 02 20:25:38.883109 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.883085 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff32b88bd49ab785dfe574e2a62b5b4f8601b466cb7cdeacfd73502f06f12b70" Nov 02 20:25:38.883362 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:38.883346 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5" Nov 02 20:25:44.761916 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:25:44.761877 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:25:51.284319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.284269 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4"] Nov 02 20:25:51.335730 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.335695 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gksh4"] Nov 02 20:25:51.392579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.392540 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm"] Nov 02 20:25:51.392990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.392977 5850 cpu_manager.go:414] "containerMap: removing container" podUID="f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f" containerName="revoke-keys" Nov 02 20:25:51.393049 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.392991 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f" containerName="revoke-keys" Nov 02 20:25:51.393090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.393085 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f" containerName="revoke-keys" Nov 02 20:25:51.396434 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.396413 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.431342 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.431305 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlfjm\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.431560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.431393 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xkjz\" (UniqueName: \"kubernetes.io/projected/e90a4cfb-3b6b-442b-8ef4-21816b72228c-kube-api-access-9xkjz\") pod \"maas-api-revoke-sub-3d4156f1-qlfjm\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.431560 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.431488 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlfjm\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.451582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.451546 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:25:51.451582 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.451552 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:25:51.452732 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.452704 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm"] Nov 02 20:25:51.532609 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.532569 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xkjz\" (UniqueName: \"kubernetes.io/projected/e90a4cfb-3b6b-442b-8ef4-21816b72228c-kube-api-access-9xkjz\") pod \"maas-api-revoke-sub-3d4156f1-qlfjm\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.532808 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.532620 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlfjm\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.532808 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.532677 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlfjm\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.533423 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.533401 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlfjm\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.535186 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.535125 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qlfjm\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.594980 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.594949 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xkjz\" (UniqueName: \"kubernetes.io/projected/e90a4cfb-3b6b-442b-8ef4-21816b72228c-kube-api-access-9xkjz\") pod \"maas-api-revoke-sub-3d4156f1-qlfjm\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.710437 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.710396 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:51.825201 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:51.825112 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942bb383-1bd6-4bc5-8ca4-73f9382e49ed" path="/var/lib/kubelet/pods/942bb383-1bd6-4bc5-8ca4-73f9382e49ed/volumes" Nov 02 20:25:52.079268 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:25:52.079232 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90a4cfb_3b6b_442b_8ef4_21816b72228c.slice/crio-89fccabe8066c436883ecb466124f7cc4c6e431713b74103733f56e0b4a0aac7 WatchSource:0}: Error finding container 89fccabe8066c436883ecb466124f7cc4c6e431713b74103733f56e0b4a0aac7: Status 404 returned error can't find the container with id 89fccabe8066c436883ecb466124f7cc4c6e431713b74103733f56e0b4a0aac7 Nov 02 20:25:52.081681 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:52.081650 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm"] Nov 02 20:25:52.841533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:52.841412 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:52.959360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:52.959324 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" event={"ID":"e90a4cfb-3b6b-442b-8ef4-21816b72228c","Type":"ContainerStarted","Data":"61a167ce4c04d9cd85ade46658415541db4d0fc8cd6fd11e046b366c91073bf9"} Nov 02 20:25:52.959579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:52.959379 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" event={"ID":"e90a4cfb-3b6b-442b-8ef4-21816b72228c","Type":"ContainerStarted","Data":"89fccabe8066c436883ecb466124f7cc4c6e431713b74103733f56e0b4a0aac7"} Nov 02 20:25:53.066673 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:53.066633 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg"] Nov 02 20:25:53.068397 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:53.068376 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-965xg"] Nov 02 20:25:53.098751 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:53.098637 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" podStartSLOduration=2.098623551 podStartE2EDuration="2.098623551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:53.097362398 +0000 UTC m=+3411.439751473" watchObservedRunningTime="2026-11-02 20:25:53.098623551 +0000 UTC m=+3411.441012631" Nov 02 20:25:53.457671 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:53.457628 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:53.813619 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:53.813536 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8c888a-390b-4ea3-ab52-7a8672c6942d" path="/var/lib/kubelet/pods/2a8c888a-390b-4ea3-ab52-7a8672c6942d/volumes" Nov 02 20:25:53.965421 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:53.965387 5850 generic.go:356] "Generic (PLEG): container finished" podID="e90a4cfb-3b6b-442b-8ef4-21816b72228c" containerID="61a167ce4c04d9cd85ade46658415541db4d0fc8cd6fd11e046b366c91073bf9" exitCode=0 Nov 02 20:25:53.965859 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:53.965485 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" event={"ID":"e90a4cfb-3b6b-442b-8ef4-21816b72228c","Type":"ContainerDied","Data":"61a167ce4c04d9cd85ade46658415541db4d0fc8cd6fd11e046b366c91073bf9"} Nov 02 20:25:55.115280 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.115251 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:55.169899 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.169862 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-xks-ca\") pod \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " Nov 02 20:25:55.169899 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.169902 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90a4cfb-3b6b-442b-8ef4-21816b72228c-kube-api-access-9xkjz\" (UniqueName: \"kubernetes.io/projected/e90a4cfb-3b6b-442b-8ef4-21816b72228c-kube-api-access-9xkjz\") pod \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " Nov 02 20:25:55.170120 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.169999 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-openshift-ca\") pod \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\" (UID: \"e90a4cfb-3b6b-442b-8ef4-21816b72228c\") " Nov 02 20:25:55.170404 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.170381 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-openshift-ca" pod "e90a4cfb-3b6b-442b-8ef4-21816b72228c" (UID: "e90a4cfb-3b6b-442b-8ef4-21816b72228c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:55.172066 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.172037 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-xks-ca" pod "e90a4cfb-3b6b-442b-8ef4-21816b72228c" (UID: "e90a4cfb-3b6b-442b-8ef4-21816b72228c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:25:55.172178 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.172100 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90a4cfb-3b6b-442b-8ef4-21816b72228c-kube-api-access-9xkjz" pod "e90a4cfb-3b6b-442b-8ef4-21816b72228c" (UID: "e90a4cfb-3b6b-442b-8ef4-21816b72228c"). InnerVolumeSpecName "kube-api-access-9xkjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:55.271045 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.270957 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:55.271045 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.270992 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a4cfb-3b6b-442b-8ef4-21816b72228c-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:55.271045 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.271005 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xkjz\" (UniqueName: \"kubernetes.io/projected/e90a4cfb-3b6b-442b-8ef4-21816b72228c-kube-api-access-9xkjz\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:55.978017 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.977984 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" Nov 02 20:25:55.978213 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.978016 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm" event={"ID":"e90a4cfb-3b6b-442b-8ef4-21816b72228c","Type":"ContainerDied","Data":"89fccabe8066c436883ecb466124f7cc4c6e431713b74103733f56e0b4a0aac7"} Nov 02 20:25:55.978213 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:25:55.978064 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89fccabe8066c436883ecb466124f7cc4c6e431713b74103733f56e0b4a0aac7" Nov 02 20:26:03.704915 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:03.704882 5850 scope.go:122] "RemoveContainer" containerID="890972bde7ae35db04b3fc217511ed72ed950c5fd3370579024b65ae1cdb9f6a" Nov 02 20:26:07.373684 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:07.373642 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:08.015999 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:08.015961 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:08.284206 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:08.284117 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:12.129262 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:12.129154 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:12.289807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:12.289769 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:12.461415 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:12.461290 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:14.760972 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:26:14.760937 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:26:15.418613 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:15.418577 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:15.594024 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:15.593975 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:15.953043 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:15.953008 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np"] Nov 02 20:26:15.953452 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:15.953439 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e90a4cfb-3b6b-442b-8ef4-21816b72228c" containerName="revoke-keys" Nov 02 20:26:15.953537 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:15.953453 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90a4cfb-3b6b-442b-8ef4-21816b72228c" containerName="revoke-keys" Nov 02 20:26:15.953627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:15.953562 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90a4cfb-3b6b-442b-8ef4-21816b72228c" containerName="revoke-keys" Nov 02 20:26:15.958303 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:15.958276 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:16.004264 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.004223 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ccmj\" (UniqueName: \"kubernetes.io/projected/339cb77a-dd17-40fe-b943-89d3fca902b2-kube-api-access-2ccmj\") pod \"payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np\" (UID: \"339cb77a-dd17-40fe-b943-89d3fca902b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:16.004452 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.004274 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339cb77a-dd17-40fe-b943-89d3fca902b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np\" (UID: \"339cb77a-dd17-40fe-b943-89d3fca902b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:16.015418 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.015377 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np"] Nov 02 20:26:16.063846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.063812 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ef8d045f-dockercfg-f8nm7\"" Nov 02 20:26:16.063846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.063816 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-9ff720\"" Nov 02 20:26:16.105489 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.105439 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ccmj\" (UniqueName: \"kubernetes.io/projected/339cb77a-dd17-40fe-b943-89d3fca902b2-kube-api-access-2ccmj\") pod \"payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np\" (UID: \"339cb77a-dd17-40fe-b943-89d3fca902b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:16.105682 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.105510 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339cb77a-dd17-40fe-b943-89d3fca902b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np\" (UID: \"339cb77a-dd17-40fe-b943-89d3fca902b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:16.106197 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.106176 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339cb77a-dd17-40fe-b943-89d3fca902b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np\" (UID: \"339cb77a-dd17-40fe-b943-89d3fca902b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:16.164312 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.164267 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ccmj\" (UniqueName: \"kubernetes.io/projected/339cb77a-dd17-40fe-b943-89d3fca902b2-kube-api-access-2ccmj\") pod \"payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np\" (UID: \"339cb77a-dd17-40fe-b943-89d3fca902b2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:16.271784 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.271684 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:16.644941 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:26:16.644900 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339cb77a_dd17_40fe_b943_89d3fca902b2.slice/crio-d0c73c079d7c738f5b3265b6556b7eb80a1d8a9d04b016a0d6f3df6b503db834 WatchSource:0}: Error finding container d0c73c079d7c738f5b3265b6556b7eb80a1d8a9d04b016a0d6f3df6b503db834: Status 404 returned error can't find the container with id d0c73c079d7c738f5b3265b6556b7eb80a1d8a9d04b016a0d6f3df6b503db834 Nov 02 20:26:16.646233 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:16.646205 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np"] Nov 02 20:26:17.090628 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:17.090513 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" event={"ID":"339cb77a-dd17-40fe-b943-89d3fca902b2","Type":"ContainerStarted","Data":"0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2"} Nov 02 20:26:17.090628 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:17.090591 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" event={"ID":"339cb77a-dd17-40fe-b943-89d3fca902b2","Type":"ContainerStarted","Data":"d0c73c079d7c738f5b3265b6556b7eb80a1d8a9d04b016a0d6f3df6b503db834"} Nov 02 20:26:17.090628 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:17.090616 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:17.208612 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:17.208555 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" podStartSLOduration=2.208539696 podStartE2EDuration="2.208539696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:17.207789586 +0000 UTC m=+3435.550178666" watchObservedRunningTime="2026-11-02 20:26:17.208539696 +0000 UTC m=+3435.550928781" Nov 02 20:26:17.978723 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:17.978687 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:18.129149 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:18.129116 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:18.286501 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:18.286377 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:20.721585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:20.721554 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:20.874767 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:20.874731 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:22.447119 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:22.447077 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:22.599291 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:22.599260 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:26.050962 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:26.050920 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw"] Nov 02 20:26:26.102181 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:26.102148 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dcmrw"] Nov 02 20:26:26.950743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:26.950712 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:27.814297 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:27.814265 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cddf633d-93a4-4c5d-98df-9c6113d82271" path="/var/lib/kubelet/pods/cddf633d-93a4-4c5d-98df-9c6113d82271/volumes" Nov 02 20:26:28.100248 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:28.100160 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:26:31.360646 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:31.360567 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:35.960333 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:35.960290 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:36.995494 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:36.995442 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:37.583359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:37.583320 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:37.740313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:37.740280 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:37.900266 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:37.900232 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:38.054984 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:38.054945 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w"] Nov 02 20:26:38.106413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:38.106379 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2w56w"] Nov 02 20:26:39.813763 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:39.813722 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22c9791-218a-4482-ac79-9122fc7e3b05" path="/var/lib/kubelet/pods/e22c9791-218a-4482-ac79-9122fc7e3b05/volumes" Nov 02 20:26:40.337370 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:40.337336 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:41.284281 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:41.284245 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:42.631393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:42.631354 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:44.761138 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:26:44.761105 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:26:44.837154 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:44.837119 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:49.233958 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:49.233926 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:53.629691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:53.629655 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:57.959944 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:26:57.959912 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:02.329942 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:02.329907 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:03.727285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:03.727242 5850 scope.go:122] "RemoveContainer" containerID="924007d7d45c98b5dea74b68aa68086107c562a872c2f1ce2cd647eb722bdd28" Nov 02 20:27:06.764718 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:06.761519 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:11.254235 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:11.254203 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:14.760588 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:27:14.760548 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:27:15.631748 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:15.631712 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:16.045981 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:16.045897 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs"] Nov 02 20:27:16.097150 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:16.097112 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m7dfs"] Nov 02 20:27:17.814535 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:17.814502 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df06cda2-38e9-4d09-aa3f-e0f705f0aa4d" path="/var/lib/kubelet/pods/df06cda2-38e9-4d09-aa3f-e0f705f0aa4d/volumes" Nov 02 20:27:20.052549 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:20.052511 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:22.810835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:22.810803 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:23.295993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:23.295940 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:24.116580 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:24.116525 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:24.425437 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:24.425329 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:25.340059 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.340014 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48"] Nov 02 20:27:25.345035 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.345007 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.400353 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.400316 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48"] Nov 02 20:27:25.431142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.431107 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:27:25.431386 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.431372 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:27:25.435914 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.435889 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8q8\" (UniqueName: \"kubernetes.io/projected/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-kube-api-access-9f8q8\") pod \"maas-api-revoke-sub-f118e8d9-prf48\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.435994 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.435951 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f118e8d9-prf48\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.436054 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.436042 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f118e8d9-prf48\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.537366 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.537328 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8q8\" (UniqueName: \"kubernetes.io/projected/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-kube-api-access-9f8q8\") pod \"maas-api-revoke-sub-f118e8d9-prf48\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.537600 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.537521 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f118e8d9-prf48\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.537672 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.537612 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f118e8d9-prf48\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.538282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.538261 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f118e8d9-prf48\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.540095 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.540063 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f118e8d9-prf48\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.598013 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.597938 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8q8\" (UniqueName: \"kubernetes.io/projected/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-kube-api-access-9f8q8\") pod \"maas-api-revoke-sub-f118e8d9-prf48\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:25.657904 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:25.657854 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:26.028810 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:27:26.028771 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260b6bd5_deaf_4879_9ee7_0cb6a2cfcd98.slice/crio-6ee8c20ee1fb1ab141cd7a7d8803882345e3b6f24dc792305cfdd0922983e4b3 WatchSource:0}: Error finding container 6ee8c20ee1fb1ab141cd7a7d8803882345e3b6f24dc792305cfdd0922983e4b3: Status 404 returned error can't find the container with id 6ee8c20ee1fb1ab141cd7a7d8803882345e3b6f24dc792305cfdd0922983e4b3 Nov 02 20:27:26.030850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:26.030825 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48"] Nov 02 20:27:26.485081 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:26.485039 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" event={"ID":"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98","Type":"ContainerStarted","Data":"fef3b215d8461f904d0ea9224cf23249686dbd89349822e28fd53be2e5e0bd6a"} Nov 02 20:27:26.485533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:26.485107 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" event={"ID":"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98","Type":"ContainerStarted","Data":"6ee8c20ee1fb1ab141cd7a7d8803882345e3b6f24dc792305cfdd0922983e4b3"} Nov 02 20:27:26.656542 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:26.656441 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" podStartSLOduration=1.656425351 podStartE2EDuration="1.656425351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:26.604683304 +0000 UTC m=+3504.947072383" watchObservedRunningTime="2026-11-02 20:27:26.656425351 +0000 UTC m=+3504.998814431" Nov 02 20:27:27.046493 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:27.046379 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh"] Nov 02 20:27:27.050338 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:27.050308 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bccsh"] Nov 02 20:27:27.491928 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:27.491886 5850 generic.go:356] "Generic (PLEG): container finished" podID="260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98" containerID="fef3b215d8461f904d0ea9224cf23249686dbd89349822e28fd53be2e5e0bd6a" exitCode=0 Nov 02 20:27:27.492353 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:27.491964 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" event={"ID":"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98","Type":"ContainerDied","Data":"fef3b215d8461f904d0ea9224cf23249686dbd89349822e28fd53be2e5e0bd6a"} Nov 02 20:27:27.814517 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:27.814419 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3340609-1895-4a4e-8dee-4319e1fecac0" path="/var/lib/kubelet/pods/d3340609-1895-4a4e-8dee-4319e1fecac0/volumes" Nov 02 20:27:28.632910 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.632881 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:28.670976 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.670883 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-kube-api-access-9f8q8\" (UniqueName: \"kubernetes.io/projected/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-kube-api-access-9f8q8\") pod \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " Nov 02 20:27:28.671168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.671067 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-openshift-ca\") pod \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " Nov 02 20:27:28.671168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.671105 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-xks-ca\") pod \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\" (UID: \"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98\") " Nov 02 20:27:28.671464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.671437 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-openshift-ca" pod "260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98" (UID: "260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:27:28.673097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.673068 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-kube-api-access-9f8q8" pod "260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98" (UID: "260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98"). InnerVolumeSpecName "kube-api-access-9f8q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:27:28.673215 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.673135 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-xks-ca" pod "260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98" (UID: "260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:27:28.772797 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.772698 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8q8\" (UniqueName: \"kubernetes.io/projected/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-kube-api-access-9f8q8\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:28.772797 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.772747 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:28.772797 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.772764 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:28.847656 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:28.847621 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:29.503960 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:29.503930 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" Nov 02 20:27:29.504244 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:29.503934 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48" event={"ID":"260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98","Type":"ContainerDied","Data":"6ee8c20ee1fb1ab141cd7a7d8803882345e3b6f24dc792305cfdd0922983e4b3"} Nov 02 20:27:29.504244 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:29.504061 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ee8c20ee1fb1ab141cd7a7d8803882345e3b6f24dc792305cfdd0922983e4b3" Nov 02 20:27:33.220333 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:33.220292 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:35.191532 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:35.191496 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:37.008936 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.008899 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:37.511412 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.511370 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9"] Nov 02 20:27:37.511850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.511836 5850 cpu_manager.go:414] "containerMap: removing container" podUID="260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98" containerName="revoke-keys" Nov 02 20:27:37.511904 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.511851 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98" containerName="revoke-keys" Nov 02 20:27:37.511942 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.511928 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98" containerName="revoke-keys" Nov 02 20:27:37.514790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.514767 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.570126 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.570088 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jphm\" (UniqueName: \"kubernetes.io/projected/3d743af9-2e2e-455a-9f47-1d451f11267d-kube-api-access-4jphm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.570345 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.570139 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.570345 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.570257 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:27:37.570516 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.570344 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.621533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.621493 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:27:37.623452 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.623426 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9"] Nov 02 20:27:37.671303 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.671261 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.671526 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.671363 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jphm\" (UniqueName: \"kubernetes.io/projected/3d743af9-2e2e-455a-9f47-1d451f11267d-kube-api-access-4jphm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.671526 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.671399 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.672112 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.672083 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.673808 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.673787 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.717986 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.717950 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:37.729590 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.729559 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jphm\" (UniqueName: \"kubernetes.io/projected/3d743af9-2e2e-455a-9f47-1d451f11267d-kube-api-access-4jphm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:37.827545 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:37.827412 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:38.194706 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:27:38.194662 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d743af9_2e2e_455a_9f47_1d451f11267d.slice/crio-68eb9de3f9347a9446610bd1bc43138677d6d99137213e3db3a3932977ca3a2b WatchSource:0}: Error finding container 68eb9de3f9347a9446610bd1bc43138677d6d99137213e3db3a3932977ca3a2b: Status 404 returned error can't find the container with id 68eb9de3f9347a9446610bd1bc43138677d6d99137213e3db3a3932977ca3a2b Nov 02 20:27:38.195973 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:38.195946 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9"] Nov 02 20:27:38.553416 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:38.553376 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" event={"ID":"3d743af9-2e2e-455a-9f47-1d451f11267d","Type":"ContainerStarted","Data":"a6cd1aa79f89e55fc43d46284f1c7e3307f2945c44be0a490f7fc5ded22ef1b0"} Nov 02 20:27:38.553613 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:38.553440 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" event={"ID":"3d743af9-2e2e-455a-9f47-1d451f11267d","Type":"ContainerStarted","Data":"68eb9de3f9347a9446610bd1bc43138677d6d99137213e3db3a3932977ca3a2b"} Nov 02 20:27:38.669336 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:38.669284 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" podStartSLOduration=1.669271433 podStartE2EDuration="1.669271433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:38.668453386 +0000 UTC m=+3517.010842465" watchObservedRunningTime="2026-11-02 20:27:38.669271433 +0000 UTC m=+3517.011660562" Nov 02 20:27:39.560631 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:39.560596 5850 generic.go:356] "Generic (PLEG): container finished" podID="3d743af9-2e2e-455a-9f47-1d451f11267d" containerID="a6cd1aa79f89e55fc43d46284f1c7e3307f2945c44be0a490f7fc5ded22ef1b0" exitCode=0 Nov 02 20:27:39.561076 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:39.560690 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" event={"ID":"3d743af9-2e2e-455a-9f47-1d451f11267d","Type":"ContainerDied","Data":"a6cd1aa79f89e55fc43d46284f1c7e3307f2945c44be0a490f7fc5ded22ef1b0"} Nov 02 20:27:40.704758 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.704725 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:40.805102 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.805052 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-xks-ca\") pod \"3d743af9-2e2e-455a-9f47-1d451f11267d\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " Nov 02 20:27:40.805312 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.805132 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d743af9-2e2e-455a-9f47-1d451f11267d-kube-api-access-4jphm\" (UniqueName: \"kubernetes.io/projected/3d743af9-2e2e-455a-9f47-1d451f11267d-kube-api-access-4jphm\") pod \"3d743af9-2e2e-455a-9f47-1d451f11267d\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " Nov 02 20:27:40.805312 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.805193 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-openshift-ca\") pod \"3d743af9-2e2e-455a-9f47-1d451f11267d\" (UID: \"3d743af9-2e2e-455a-9f47-1d451f11267d\") " Nov 02 20:27:40.805572 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.805547 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-openshift-ca" pod "3d743af9-2e2e-455a-9f47-1d451f11267d" (UID: "3d743af9-2e2e-455a-9f47-1d451f11267d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:27:40.807324 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.807300 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d743af9-2e2e-455a-9f47-1d451f11267d-kube-api-access-4jphm" pod "3d743af9-2e2e-455a-9f47-1d451f11267d" (UID: "3d743af9-2e2e-455a-9f47-1d451f11267d"). InnerVolumeSpecName "kube-api-access-4jphm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:27:40.807400 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.807375 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-xks-ca" pod "3d743af9-2e2e-455a-9f47-1d451f11267d" (UID: "3d743af9-2e2e-455a-9f47-1d451f11267d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:27:40.906731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.906676 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:40.906731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.906725 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jphm\" (UniqueName: \"kubernetes.io/projected/3d743af9-2e2e-455a-9f47-1d451f11267d-kube-api-access-4jphm\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:40.906731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:40.906738 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d743af9-2e2e-455a-9f47-1d451f11267d-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:41.572483 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:41.572441 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" Nov 02 20:27:41.572704 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:41.572503 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9" event={"ID":"3d743af9-2e2e-455a-9f47-1d451f11267d","Type":"ContainerDied","Data":"68eb9de3f9347a9446610bd1bc43138677d6d99137213e3db3a3932977ca3a2b"} Nov 02 20:27:41.572704 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:41.572550 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68eb9de3f9347a9446610bd1bc43138677d6d99137213e3db3a3932977ca3a2b" Nov 02 20:27:42.059238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:42.059203 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:42.538692 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:42.538650 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:44.761468 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:27:44.761427 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:27:46.935044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:46.935003 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:49.093846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:49.093809 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np"] Nov 02 20:27:49.535190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:49.535149 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:49.693062 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:49.693029 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:51.319408 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:51.319369 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:54.094457 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:54.094387 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" podUID="339cb77a-dd17-40fe-b943-89d3fca902b2" containerName="payload-processing" containerID="cri-o://0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2" gracePeriod=25 Nov 02 20:27:55.163761 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:55.163725 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:56.540368 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:56.540330 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:56.804527 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:56.804421 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:57.077152 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:57.077055 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9"] Nov 02 20:27:57.128259 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:57.128217 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9ff720-x5dm9"] Nov 02 20:27:57.568689 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:57.568660 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:57.813973 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:57.813941 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d743af9-2e2e-455a-9f47-1d451f11267d" path="/var/lib/kubelet/pods/3d743af9-2e2e-455a-9f47-1d451f11267d/volumes" Nov 02 20:27:58.098330 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:27:58.098287 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" podUID="339cb77a-dd17-40fe-b943-89d3fca902b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:28:01.046923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:01.046888 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6"] Nov 02 20:28:01.101676 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:01.101639 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qlgl6"] Nov 02 20:28:01.814319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:01.814288 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7978678-a282-4214-9240-bb0cd1e910b6" path="/var/lib/kubelet/pods/d7978678-a282-4214-9240-bb0cd1e910b6/volumes" Nov 02 20:28:03.262446 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:03.262405 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:03.421770 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:03.421739 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:03.623447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:03.623411 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:03.752780 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:03.752743 5850 scope.go:122] "RemoveContainer" containerID="9eedfa1cf1ba4eeba48adff9044cf6f77c75bf20a3744f569add00fdce160109" Nov 02 20:28:03.763044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:03.763019 5850 scope.go:122] "RemoveContainer" containerID="6be66a3b254dd9f8718fc0049e61e90d28a4bfdc32ce0e494f1fc23d16274157" Nov 02 20:28:05.275081 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.275042 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm"] Nov 02 20:28:05.326593 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.326552 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qlfjm"] Nov 02 20:28:05.387548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.387513 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n"] Nov 02 20:28:05.387977 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.387963 5850 cpu_manager.go:414] "containerMap: removing container" podUID="3d743af9-2e2e-455a-9f47-1d451f11267d" containerName="revoke-keys" Nov 02 20:28:05.388022 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.387979 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d743af9-2e2e-455a-9f47-1d451f11267d" containerName="revoke-keys" Nov 02 20:28:05.388069 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.388058 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d743af9-2e2e-455a-9f47-1d451f11267d" containerName="revoke-keys" Nov 02 20:28:05.392233 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.392210 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.446809 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.446753 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:28:05.447148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.447120 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:28:05.449387 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.449362 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n"] Nov 02 20:28:05.458569 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.458532 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27n\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.458737 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.458645 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpw6t\" (UniqueName: \"kubernetes.io/projected/66e3bb96-c06a-4c9e-b667-a36c18372c26-kube-api-access-zpw6t\") pod \"maas-api-revoke-sub-3d4156f1-nh27n\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.458783 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.458764 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27n\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.560039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.559932 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpw6t\" (UniqueName: \"kubernetes.io/projected/66e3bb96-c06a-4c9e-b667-a36c18372c26-kube-api-access-zpw6t\") pod \"maas-api-revoke-sub-3d4156f1-nh27n\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.560039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.560039 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27n\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.560314 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.560078 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27n\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.560756 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.560729 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27n\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.562521 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.562494 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27n\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.618669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.618629 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpw6t\" (UniqueName: \"kubernetes.io/projected/66e3bb96-c06a-4c9e-b667-a36c18372c26-kube-api-access-zpw6t\") pod \"maas-api-revoke-sub-3d4156f1-nh27n\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.705913 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.705874 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:05.814579 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:05.814493 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90a4cfb-3b6b-442b-8ef4-21816b72228c" path="/var/lib/kubelet/pods/e90a4cfb-3b6b-442b-8ef4-21816b72228c/volumes" Nov 02 20:28:06.073357 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:28:06.073301 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e3bb96_c06a_4c9e_b667_a36c18372c26.slice/crio-ea52fe1ab434c4236ab99e11af01ca353a6708c2826b7817d51deb3ad1eaa341 WatchSource:0}: Error finding container ea52fe1ab434c4236ab99e11af01ca353a6708c2826b7817d51deb3ad1eaa341: Status 404 returned error can't find the container with id ea52fe1ab434c4236ab99e11af01ca353a6708c2826b7817d51deb3ad1eaa341 Nov 02 20:28:06.076025 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:06.075997 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n"] Nov 02 20:28:06.724783 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:06.724747 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" event={"ID":"66e3bb96-c06a-4c9e-b667-a36c18372c26","Type":"ContainerStarted","Data":"ee08e0824b9270f575980917f51daced2df9716f10cdd505505ae4668a0bbedd"} Nov 02 20:28:06.725190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:06.724803 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" event={"ID":"66e3bb96-c06a-4c9e-b667-a36c18372c26","Type":"ContainerStarted","Data":"ea52fe1ab434c4236ab99e11af01ca353a6708c2826b7817d51deb3ad1eaa341"} Nov 02 20:28:06.842660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:06.842596 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" podStartSLOduration=1.8425822539999999 podStartE2EDuration="1.842582254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:06.842320004 +0000 UTC m=+3545.184709095" watchObservedRunningTime="2026-11-02 20:28:06.842582254 +0000 UTC m=+3545.184971328" Nov 02 20:28:07.730946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:07.730908 5850 generic.go:356] "Generic (PLEG): container finished" podID="66e3bb96-c06a-4c9e-b667-a36c18372c26" containerID="ee08e0824b9270f575980917f51daced2df9716f10cdd505505ae4668a0bbedd" exitCode=0 Nov 02 20:28:07.731351 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:07.730995 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" event={"ID":"66e3bb96-c06a-4c9e-b667-a36c18372c26","Type":"ContainerDied","Data":"ee08e0824b9270f575980917f51daced2df9716f10cdd505505ae4668a0bbedd"} Nov 02 20:28:08.098779 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:08.098682 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" podUID="339cb77a-dd17-40fe-b943-89d3fca902b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:28:08.871441 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:08.871414 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:08.994933 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:08.994853 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-xks-ca\") pod \"66e3bb96-c06a-4c9e-b667-a36c18372c26\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " Nov 02 20:28:08.994933 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:08.994909 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e3bb96-c06a-4c9e-b667-a36c18372c26-kube-api-access-zpw6t\" (UniqueName: \"kubernetes.io/projected/66e3bb96-c06a-4c9e-b667-a36c18372c26-kube-api-access-zpw6t\") pod \"66e3bb96-c06a-4c9e-b667-a36c18372c26\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " Nov 02 20:28:08.995134 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:08.994946 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-openshift-ca\") pod \"66e3bb96-c06a-4c9e-b667-a36c18372c26\" (UID: \"66e3bb96-c06a-4c9e-b667-a36c18372c26\") " Nov 02 20:28:08.995384 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:08.995354 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-openshift-ca" pod "66e3bb96-c06a-4c9e-b667-a36c18372c26" (UID: "66e3bb96-c06a-4c9e-b667-a36c18372c26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:08.997229 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:08.997196 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e3bb96-c06a-4c9e-b667-a36c18372c26-kube-api-access-zpw6t" pod "66e3bb96-c06a-4c9e-b667-a36c18372c26" (UID: "66e3bb96-c06a-4c9e-b667-a36c18372c26"). InnerVolumeSpecName "kube-api-access-zpw6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:08.997389 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:08.997245 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-xks-ca" pod "66e3bb96-c06a-4c9e-b667-a36c18372c26" (UID: "66e3bb96-c06a-4c9e-b667-a36c18372c26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:09.096786 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:09.096751 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:09.096786 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:09.096782 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpw6t\" (UniqueName: \"kubernetes.io/projected/66e3bb96-c06a-4c9e-b667-a36c18372c26-kube-api-access-zpw6t\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:09.096786 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:09.096792 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e3bb96-c06a-4c9e-b667-a36c18372c26-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:09.743923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:09.743894 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" Nov 02 20:28:09.744111 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:09.743927 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n" event={"ID":"66e3bb96-c06a-4c9e-b667-a36c18372c26","Type":"ContainerDied","Data":"ea52fe1ab434c4236ab99e11af01ca353a6708c2826b7817d51deb3ad1eaa341"} Nov 02 20:28:09.744111 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:09.743969 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea52fe1ab434c4236ab99e11af01ca353a6708c2826b7817d51deb3ad1eaa341" Nov 02 20:28:14.346803 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.346774 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:28:14.448388 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.448346 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339cb77a-dd17-40fe-b943-89d3fca902b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339cb77a-dd17-40fe-b943-89d3fca902b2-plugins-config-volume\") pod \"339cb77a-dd17-40fe-b943-89d3fca902b2\" (UID: \"339cb77a-dd17-40fe-b943-89d3fca902b2\") " Nov 02 20:28:14.448622 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.448532 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339cb77a-dd17-40fe-b943-89d3fca902b2-kube-api-access-2ccmj\" (UniqueName: \"kubernetes.io/projected/339cb77a-dd17-40fe-b943-89d3fca902b2-kube-api-access-2ccmj\") pod \"339cb77a-dd17-40fe-b943-89d3fca902b2\" (UID: \"339cb77a-dd17-40fe-b943-89d3fca902b2\") " Nov 02 20:28:14.448882 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.448852 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339cb77a-dd17-40fe-b943-89d3fca902b2-plugins-config-volume" pod "339cb77a-dd17-40fe-b943-89d3fca902b2" (UID: "339cb77a-dd17-40fe-b943-89d3fca902b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:14.450691 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.450666 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339cb77a-dd17-40fe-b943-89d3fca902b2-kube-api-access-2ccmj" pod "339cb77a-dd17-40fe-b943-89d3fca902b2" (UID: "339cb77a-dd17-40fe-b943-89d3fca902b2"). InnerVolumeSpecName "kube-api-access-2ccmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:14.549628 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.549585 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339cb77a-dd17-40fe-b943-89d3fca902b2-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:14.549628 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.549621 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ccmj\" (UniqueName: \"kubernetes.io/projected/339cb77a-dd17-40fe-b943-89d3fca902b2-kube-api-access-2ccmj\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:14.761494 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:28:14.761397 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:28:14.770536 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.770386 5850 generic.go:356] "Generic (PLEG): container finished" podID="339cb77a-dd17-40fe-b943-89d3fca902b2" containerID="0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2" exitCode=0 Nov 02 20:28:14.770536 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.770429 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" Nov 02 20:28:14.770536 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.770489 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" event={"ID":"339cb77a-dd17-40fe-b943-89d3fca902b2","Type":"ContainerDied","Data":"0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2"} Nov 02 20:28:14.789393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.770556 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np" event={"ID":"339cb77a-dd17-40fe-b943-89d3fca902b2","Type":"ContainerDied","Data":"d0c73c079d7c738f5b3265b6556b7eb80a1d8a9d04b016a0d6f3df6b503db834"} Nov 02 20:28:14.789393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.770587 5850 scope.go:122] "RemoveContainer" containerID="0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2" Nov 02 20:28:14.789393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.780591 5850 scope.go:122] "RemoveContainer" containerID="0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2" Nov 02 20:28:14.789393 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:28:14.780894 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2\": container with ID starting with 0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2 not found: ID does not exist" containerID="0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2" Nov 02 20:28:14.789393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.780923 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2"} err="failed to get container status \"0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2\": rpc error: code = NotFound desc = could not find container \"0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2\": container with ID starting with 0a444e12c479286c05696a6126234e98d342976453d7f746592f9e1f48597ef2 not found: ID does not exist" Nov 02 20:28:14.939282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.939242 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np"] Nov 02 20:28:14.940780 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:14.940756 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ff720-5cb864bbcp28np"] Nov 02 20:28:15.813750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:15.813717 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339cb77a-dd17-40fe-b943-89d3fca902b2" path="/var/lib/kubelet/pods/339cb77a-dd17-40fe-b943-89d3fca902b2/volumes" Nov 02 20:28:18.769735 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:18.769698 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:18.987104 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:18.987067 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:23.106531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:23.106461 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:23.444046 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:23.444015 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:24.418671 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:24.418636 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:29.046548 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:29.046507 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx"] Nov 02 20:28:29.097919 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:29.097881 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vzkxx"] Nov 02 20:28:29.814113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:29.814078 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1603608e-16f2-47b2-b3e0-ae60a4f1699c" path="/var/lib/kubelet/pods/1603608e-16f2-47b2-b3e0-ae60a4f1699c/volumes" Nov 02 20:28:32.063744 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:28:32.063708 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:44.760779 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:28:44.760730 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:29:00.047032 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:00.046989 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs"] Nov 02 20:29:00.098424 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:00.098387 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xcbxs"] Nov 02 20:29:01.814265 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:01.814227 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe6080b2-d5c7-472d-b895-45a98d83ecc5" path="/var/lib/kubelet/pods/fe6080b2-d5c7-472d-b895-45a98d83ecc5/volumes" Nov 02 20:29:03.789786 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:03.789753 5850 scope.go:122] "RemoveContainer" containerID="16b76553db5995e6af014169ef02054d61b309a0d4513ec982e625a25fa4eeca" Nov 02 20:29:03.798689 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:03.798663 5850 scope.go:122] "RemoveContainer" containerID="aa8ce83f4746a03e380f8b2dc9fb67147d35c9b9446fc23d31039de12708605f" Nov 02 20:29:03.807328 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:03.807303 5850 scope.go:122] "RemoveContainer" containerID="944367e2e0c0b2ce3864374a20e2861a1899e24e428c50539bdb67be41adfe19" Nov 02 20:29:14.761444 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:29:14.761365 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:29:20.046055 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:20.046014 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m"] Nov 02 20:29:20.097686 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:20.097640 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6q52m"] Nov 02 20:29:21.814198 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:21.814165 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57380394-db7f-482e-9ce5-093323f97663" path="/var/lib/kubelet/pods/57380394-db7f-482e-9ce5-093323f97663/volumes" Nov 02 20:29:23.492042 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:23.491934 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:44.760863 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:29:44.760826 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:29:56.940350 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:56.940310 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64"] Nov 02 20:29:56.940902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:56.940741 5850 cpu_manager.go:414] "containerMap: removing container" podUID="339cb77a-dd17-40fe-b943-89d3fca902b2" containerName="payload-processing" Nov 02 20:29:56.940902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:56.940754 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339cb77a-dd17-40fe-b943-89d3fca902b2" containerName="payload-processing" Nov 02 20:29:56.940902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:56.940780 5850 cpu_manager.go:414] "containerMap: removing container" podUID="66e3bb96-c06a-4c9e-b667-a36c18372c26" containerName="revoke-keys" Nov 02 20:29:56.940902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:56.940786 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e3bb96-c06a-4c9e-b667-a36c18372c26" containerName="revoke-keys" Nov 02 20:29:56.940902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:56.940857 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e3bb96-c06a-4c9e-b667-a36c18372c26" containerName="revoke-keys" Nov 02 20:29:56.940902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:56.940870 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="339cb77a-dd17-40fe-b943-89d3fca902b2" containerName="payload-processing" Nov 02 20:29:56.944139 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:56.944122 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.000890 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.000846 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64"] Nov 02 20:29:57.019931 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.019893 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-lvb64\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.020116 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.020049 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8vbb\" (UniqueName: \"kubernetes.io/projected/7eb3107e-83c7-483b-b696-e0b6e8fac49d-kube-api-access-d8vbb\") pod \"maas-api-revoke-sub-d576e68b-lvb64\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.020198 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.020118 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-lvb64\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.049683 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.049630 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:29:57.049880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.049831 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:29:57.120772 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.120725 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8vbb\" (UniqueName: \"kubernetes.io/projected/7eb3107e-83c7-483b-b696-e0b6e8fac49d-kube-api-access-d8vbb\") pod \"maas-api-revoke-sub-d576e68b-lvb64\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.120979 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.120782 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-lvb64\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.120979 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.120900 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-lvb64\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.121608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.121584 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-lvb64\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.123223 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.123201 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-lvb64\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.179060 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.179025 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8vbb\" (UniqueName: \"kubernetes.io/projected/7eb3107e-83c7-483b-b696-e0b6e8fac49d-kube-api-access-d8vbb\") pod \"maas-api-revoke-sub-d576e68b-lvb64\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.257404 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.257315 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:29:57.627874 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:29:57.627826 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb3107e_83c7_483b_b696_e0b6e8fac49d.slice/crio-e378ac99d0dfb5b0bb90a067affa12541aba644a19a638925045b62e5b890ca9 WatchSource:0}: Error finding container e378ac99d0dfb5b0bb90a067affa12541aba644a19a638925045b62e5b890ca9: Status 404 returned error can't find the container with id e378ac99d0dfb5b0bb90a067affa12541aba644a19a638925045b62e5b890ca9 Nov 02 20:29:57.629856 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:57.629827 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64"] Nov 02 20:29:58.336738 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:58.336645 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" event={"ID":"7eb3107e-83c7-483b-b696-e0b6e8fac49d","Type":"ContainerStarted","Data":"b87cc40cb9066dc7fd28028c2484c4b793143062e14fb3bc76aeee3750f283c0"} Nov 02 20:29:58.336738 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:58.336736 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" event={"ID":"7eb3107e-83c7-483b-b696-e0b6e8fac49d","Type":"ContainerStarted","Data":"e378ac99d0dfb5b0bb90a067affa12541aba644a19a638925045b62e5b890ca9"} Nov 02 20:29:58.453043 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:58.452992 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" podStartSLOduration=2.452978695 podStartE2EDuration="2.452978695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:58.452309352 +0000 UTC m=+3656.794698431" watchObservedRunningTime="2026-11-02 20:29:58.452978695 +0000 UTC m=+3656.795367792" Nov 02 20:29:59.347228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:59.347186 5850 generic.go:356] "Generic (PLEG): container finished" podID="7eb3107e-83c7-483b-b696-e0b6e8fac49d" containerID="b87cc40cb9066dc7fd28028c2484c4b793143062e14fb3bc76aeee3750f283c0" exitCode=0 Nov 02 20:29:59.347657 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:29:59.347253 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" event={"ID":"7eb3107e-83c7-483b-b696-e0b6e8fac49d","Type":"ContainerDied","Data":"b87cc40cb9066dc7fd28028c2484c4b793143062e14fb3bc76aeee3750f283c0"} Nov 02 20:30:00.491170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.491137 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:30:00.551634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.551593 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-openshift-ca\") pod \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " Nov 02 20:30:00.551802 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.551650 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-xks-ca\") pod \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " Nov 02 20:30:00.551802 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.551740 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb3107e-83c7-483b-b696-e0b6e8fac49d-kube-api-access-d8vbb\" (UniqueName: \"kubernetes.io/projected/7eb3107e-83c7-483b-b696-e0b6e8fac49d-kube-api-access-d8vbb\") pod \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\" (UID: \"7eb3107e-83c7-483b-b696-e0b6e8fac49d\") " Nov 02 20:30:00.551938 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.551917 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-openshift-ca" pod "7eb3107e-83c7-483b-b696-e0b6e8fac49d" (UID: "7eb3107e-83c7-483b-b696-e0b6e8fac49d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:00.552084 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.552054 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:00.553865 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.553833 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-xks-ca" pod "7eb3107e-83c7-483b-b696-e0b6e8fac49d" (UID: "7eb3107e-83c7-483b-b696-e0b6e8fac49d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:00.553865 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.553859 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb3107e-83c7-483b-b696-e0b6e8fac49d-kube-api-access-d8vbb" pod "7eb3107e-83c7-483b-b696-e0b6e8fac49d" (UID: "7eb3107e-83c7-483b-b696-e0b6e8fac49d"). InnerVolumeSpecName "kube-api-access-d8vbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:00.652758 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.652719 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8vbb\" (UniqueName: \"kubernetes.io/projected/7eb3107e-83c7-483b-b696-e0b6e8fac49d-kube-api-access-d8vbb\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:00.652758 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:00.652755 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb3107e-83c7-483b-b696-e0b6e8fac49d-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:01.360134 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:01.360100 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" event={"ID":"7eb3107e-83c7-483b-b696-e0b6e8fac49d","Type":"ContainerDied","Data":"e378ac99d0dfb5b0bb90a067affa12541aba644a19a638925045b62e5b890ca9"} Nov 02 20:30:01.360319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:01.360138 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64" Nov 02 20:30:01.360319 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:01.360149 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e378ac99d0dfb5b0bb90a067affa12541aba644a19a638925045b62e5b890ca9" Nov 02 20:30:03.841236 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:03.841193 5850 scope.go:122] "RemoveContainer" containerID="698358e2a8c95afea6a98d00be6dfdb1055a5e926aeec27c820ea17bfa1bb55b" Nov 02 20:30:03.850324 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:03.850301 5850 scope.go:122] "RemoveContainer" containerID="f7b693dfbb8fe36bc37bc9eac2f3e1fe0610a52a9ac9cd218b51e465158a0631" Nov 02 20:30:14.775084 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:30:14.775038 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:30:23.720493 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:23.720434 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:25.796587 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.796550 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q"] Nov 02 20:30:25.797001 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.796982 5850 cpu_manager.go:414] "containerMap: removing container" podUID="7eb3107e-83c7-483b-b696-e0b6e8fac49d" containerName="revoke-keys" Nov 02 20:30:25.797001 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.796994 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb3107e-83c7-483b-b696-e0b6e8fac49d" containerName="revoke-keys" Nov 02 20:30:25.797080 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.797067 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb3107e-83c7-483b-b696-e0b6e8fac49d" containerName="revoke-keys" Nov 02 20:30:25.800261 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.800243 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:25.871365 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.871330 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ps52q\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:25.871572 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.871406 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ps52q\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:25.871572 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.871512 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdgfn\" (UniqueName: \"kubernetes.io/projected/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-kube-api-access-cdgfn\") pod \"maas-api-revoke-sub-d2bc93bc-ps52q\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:25.905994 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.905955 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:30:25.906191 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.906040 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:30:25.907762 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.907739 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q"] Nov 02 20:30:25.972597 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.972564 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ps52q\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:25.972788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.972643 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ps52q\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:25.972788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.972674 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdgfn\" (UniqueName: \"kubernetes.io/projected/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-kube-api-access-cdgfn\") pod \"maas-api-revoke-sub-d2bc93bc-ps52q\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:25.973404 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.973380 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ps52q\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:25.975114 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:25.975084 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ps52q\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:26.031377 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:26.031342 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdgfn\" (UniqueName: \"kubernetes.io/projected/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-kube-api-access-cdgfn\") pod \"maas-api-revoke-sub-d2bc93bc-ps52q\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:26.113903 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:26.113823 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:26.480533 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:30:26.480501 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9243df2e_1a33_4f3c_99c0_09eab9fbcd80.slice/crio-1a86aabe8c7b2cac19da0ecb8b6eb68a66ca1b3845f3310ca39cc9328a225825 WatchSource:0}: Error finding container 1a86aabe8c7b2cac19da0ecb8b6eb68a66ca1b3845f3310ca39cc9328a225825: Status 404 returned error can't find the container with id 1a86aabe8c7b2cac19da0ecb8b6eb68a66ca1b3845f3310ca39cc9328a225825 Nov 02 20:30:26.482521 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:26.482495 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q"] Nov 02 20:30:26.518118 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:26.518079 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" event={"ID":"9243df2e-1a33-4f3c-99c0-09eab9fbcd80","Type":"ContainerStarted","Data":"1a86aabe8c7b2cac19da0ecb8b6eb68a66ca1b3845f3310ca39cc9328a225825"} Nov 02 20:30:27.047848 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:27.047755 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm"] Nov 02 20:30:27.050170 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:27.050138 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gffcm"] Nov 02 20:30:27.525128 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:27.525092 5850 generic.go:356] "Generic (PLEG): container finished" podID="9243df2e-1a33-4f3c-99c0-09eab9fbcd80" containerID="35099c33891a25933405ff4094222be44d2773ae5bdb644259e187aac6b2b7d0" exitCode=0 Nov 02 20:30:27.525315 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:27.525128 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" event={"ID":"9243df2e-1a33-4f3c-99c0-09eab9fbcd80","Type":"ContainerDied","Data":"35099c33891a25933405ff4094222be44d2773ae5bdb644259e187aac6b2b7d0"} Nov 02 20:30:27.813905 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:27.813822 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36489392-98d5-4216-a25a-169ff5958406" path="/var/lib/kubelet/pods/36489392-98d5-4216-a25a-169ff5958406/volumes" Nov 02 20:30:28.663866 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.663834 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:28.803852 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.803751 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-kube-api-access-cdgfn\" (UniqueName: \"kubernetes.io/projected/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-kube-api-access-cdgfn\") pod \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " Nov 02 20:30:28.803852 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.803823 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-openshift-ca\") pod \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " Nov 02 20:30:28.804093 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.803949 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-xks-ca\") pod \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\" (UID: \"9243df2e-1a33-4f3c-99c0-09eab9fbcd80\") " Nov 02 20:30:28.804199 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.804175 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-openshift-ca" pod "9243df2e-1a33-4f3c-99c0-09eab9fbcd80" (UID: "9243df2e-1a33-4f3c-99c0-09eab9fbcd80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:28.804308 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.804291 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.806029 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.806006 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-xks-ca" pod "9243df2e-1a33-4f3c-99c0-09eab9fbcd80" (UID: "9243df2e-1a33-4f3c-99c0-09eab9fbcd80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:28.806238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.806214 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-kube-api-access-cdgfn" pod "9243df2e-1a33-4f3c-99c0-09eab9fbcd80" (UID: "9243df2e-1a33-4f3c-99c0-09eab9fbcd80"). InnerVolumeSpecName "kube-api-access-cdgfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:28.904931 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.904886 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdgfn\" (UniqueName: \"kubernetes.io/projected/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-kube-api-access-cdgfn\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.904931 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:28.904924 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9243df2e-1a33-4f3c-99c0-09eab9fbcd80-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:29.535781 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:29.535744 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" Nov 02 20:30:29.535781 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:29.535771 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q" event={"ID":"9243df2e-1a33-4f3c-99c0-09eab9fbcd80","Type":"ContainerDied","Data":"1a86aabe8c7b2cac19da0ecb8b6eb68a66ca1b3845f3310ca39cc9328a225825"} Nov 02 20:30:29.536040 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:29.535822 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a86aabe8c7b2cac19da0ecb8b6eb68a66ca1b3845f3310ca39cc9328a225825" Nov 02 20:30:38.047277 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:38.047240 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5"] Nov 02 20:30:38.098624 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:38.098589 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rskm5"] Nov 02 20:30:38.100842 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:38.100765 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:38.840963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:38.840936 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:39.272531 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:39.272492 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:39.814036 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:39.814000 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f" path="/var/lib/kubelet/pods/f8fa2f8e-6acb-47a2-bfd7-844eb3bc5f9f/volumes" Nov 02 20:30:43.388024 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:43.387976 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:43.560836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:43.560806 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:43.721129 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:43.721036 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:44.760443 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:30:44.760402 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:30:46.656844 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:46.656808 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:46.834113 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:46.834078 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:47.186525 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.186486 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg"] Nov 02 20:30:47.186920 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.186906 5850 cpu_manager.go:414] "containerMap: removing container" podUID="9243df2e-1a33-4f3c-99c0-09eab9fbcd80" containerName="revoke-keys" Nov 02 20:30:47.186966 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.186921 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9243df2e-1a33-4f3c-99c0-09eab9fbcd80" containerName="revoke-keys" Nov 02 20:30:47.187024 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.187015 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="9243df2e-1a33-4f3c-99c0-09eab9fbcd80" containerName="revoke-keys" Nov 02 20:30:47.190291 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.190266 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:30:47.246850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.246817 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg"] Nov 02 20:30:47.292536 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.292452 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxfr\" (UniqueName: \"kubernetes.io/projected/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-kube-api-access-dqxfr\") pod \"payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg\" (UID: \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:30:47.292728 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.292594 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg\" (UID: \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:30:47.296035 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.296013 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6d135020-dockercfg-hj45f\"" Nov 02 20:30:47.296125 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.296025 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a52dfa\"" Nov 02 20:30:47.394178 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.394138 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxfr\" (UniqueName: \"kubernetes.io/projected/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-kube-api-access-dqxfr\") pod \"payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg\" (UID: \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:30:47.394371 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.394188 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg\" (UID: \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:30:47.394923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.394902 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg\" (UID: \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:30:47.453112 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.453037 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxfr\" (UniqueName: \"kubernetes.io/projected/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-kube-api-access-dqxfr\") pod \"payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg\" (UID: \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:30:47.504340 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.504302 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:30:47.872771 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:30:47.872741 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5878f8c9_e619_449f_a1df_04ac1b5eb6b9.slice/crio-23af5c94fd174e900e6d9c284172ef403654e1aa007e05b58968f3ddf8d28353 WatchSource:0}: Error finding container 23af5c94fd174e900e6d9c284172ef403654e1aa007e05b58968f3ddf8d28353: Status 404 returned error can't find the container with id 23af5c94fd174e900e6d9c284172ef403654e1aa007e05b58968f3ddf8d28353 Nov 02 20:30:47.874098 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:47.874072 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg"] Nov 02 20:30:48.640270 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:48.640229 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" event={"ID":"5878f8c9-e619-449f-a1df-04ac1b5eb6b9","Type":"ContainerStarted","Data":"d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6"} Nov 02 20:30:48.640534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:48.640281 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" event={"ID":"5878f8c9-e619-449f-a1df-04ac1b5eb6b9","Type":"ContainerStarted","Data":"23af5c94fd174e900e6d9c284172ef403654e1aa007e05b58968f3ddf8d28353"} Nov 02 20:30:48.640534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:48.640376 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:30:48.758168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:48.758111 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" podStartSLOduration=1.7580962310000001 podStartE2EDuration="1.758096231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:48.756135147 +0000 UTC m=+3707.098524241" watchObservedRunningTime="2026-11-02 20:30:48.758096231 +0000 UTC m=+3707.100485320" Nov 02 20:30:49.103748 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:49.103629 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:49.261875 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:49.261840 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:49.411279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:49.411246 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:53.647670 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:53.647635 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:58.076457 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:58.076326 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:59.649992 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:30:59.649959 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:31:02.447972 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:02.447932 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:03.064621 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:03.064579 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:03.653651 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:03.653611 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:03.815825 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:03.815795 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:03.879941 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:03.878264 5850 scope.go:122] "RemoveContainer" containerID="09862f03bfed52ee017c45c5b940c3cc4b8457911bddb4a1afe05e4f442c1b72" Nov 02 20:31:03.891520 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:03.891497 5850 scope.go:122] "RemoveContainer" containerID="8add7afd38d2c349a887ac7b257bdfcb27b65a64f6b2ffd573bdb46f402d326a" Nov 02 20:31:03.900206 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:03.900183 5850 scope.go:122] "RemoveContainer" containerID="9d4f5b566db36dd8ee17861eca7d219c151ff75ee7fc1c5550fc31ff223d6c37" Nov 02 20:31:06.371618 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:06.371581 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:07.144044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:07.144014 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:11.540854 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:11.540820 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:14.762199 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:31:14.762157 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:31:16.036993 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:16.036949 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:20.363876 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:20.363810 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:24.078699 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:24.078665 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:24.400140 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:24.400106 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:24.824641 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:24.824544 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:29.231132 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:29.231101 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:33.636385 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:33.636347 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:38.036639 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:38.036605 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:42.441838 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:42.441801 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:44.761301 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:31:44.761263 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:31:45.532332 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:45.532300 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:45.685769 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:45.685739 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:46.822237 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:46.822196 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:51.340162 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.340114 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:51.448503 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.448433 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j"] Nov 02 20:31:51.452178 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.452153 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.490464 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.490416 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cq9j\" (UniqueName: \"kubernetes.io/projected/c1c58a9c-aed0-4b17-b272-f5c9b6558552-kube-api-access-8cq9j\") pod \"maas-api-revoke-sub-dd61f0f8-mxz5j\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.490683 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.490624 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd61f0f8-mxz5j\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.490683 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.490659 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd61f0f8-mxz5j\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.511429 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.511396 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j"] Nov 02 20:31:51.560697 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.560661 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:51.560697 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.560696 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:31:51.591511 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.591392 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd61f0f8-mxz5j\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.591511 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.591438 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd61f0f8-mxz5j\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.591511 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.591502 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cq9j\" (UniqueName: \"kubernetes.io/projected/c1c58a9c-aed0-4b17-b272-f5c9b6558552-kube-api-access-8cq9j\") pod \"maas-api-revoke-sub-dd61f0f8-mxz5j\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.592157 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.592137 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd61f0f8-mxz5j\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.593801 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.593780 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd61f0f8-mxz5j\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.653101 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.653063 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cq9j\" (UniqueName: \"kubernetes.io/projected/c1c58a9c-aed0-4b17-b272-f5c9b6558552-kube-api-access-8cq9j\") pod \"maas-api-revoke-sub-dd61f0f8-mxz5j\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:51.764592 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:51.764541 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:52.137655 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:31:52.137610 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c58a9c_aed0_4b17_b272_f5c9b6558552.slice/crio-a640d4e150de9fc39b96e14bffac03d9d43d54305b6764f07c2829930d8fdd8e WatchSource:0}: Error finding container a640d4e150de9fc39b96e14bffac03d9d43d54305b6764f07c2829930d8fdd8e: Status 404 returned error can't find the container with id a640d4e150de9fc39b96e14bffac03d9d43d54305b6764f07c2829930d8fdd8e Nov 02 20:31:52.139331 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:52.139307 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j"] Nov 02 20:31:53.022283 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:53.022248 5850 generic.go:356] "Generic (PLEG): container finished" podID="c1c58a9c-aed0-4b17-b272-f5c9b6558552" containerID="619bb0ad3257575f9e77aa7c04908e6d8766f5464e885405823bbe830f5fe82e" exitCode=0 Nov 02 20:31:53.022784 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:53.022325 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" event={"ID":"c1c58a9c-aed0-4b17-b272-f5c9b6558552","Type":"ContainerDied","Data":"619bb0ad3257575f9e77aa7c04908e6d8766f5464e885405823bbe830f5fe82e"} Nov 02 20:31:53.022784 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:53.022416 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" event={"ID":"c1c58a9c-aed0-4b17-b272-f5c9b6558552","Type":"ContainerStarted","Data":"a640d4e150de9fc39b96e14bffac03d9d43d54305b6764f07c2829930d8fdd8e"} Nov 02 20:31:54.165407 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.165379 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:54.217925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.217888 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-openshift-ca\") pod \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " Nov 02 20:31:54.218082 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.217993 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-xks-ca\") pod \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " Nov 02 20:31:54.218082 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.218056 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c58a9c-aed0-4b17-b272-f5c9b6558552-kube-api-access-8cq9j\" (UniqueName: \"kubernetes.io/projected/c1c58a9c-aed0-4b17-b272-f5c9b6558552-kube-api-access-8cq9j\") pod \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\" (UID: \"c1c58a9c-aed0-4b17-b272-f5c9b6558552\") " Nov 02 20:31:54.218366 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.218330 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-openshift-ca" pod "c1c58a9c-aed0-4b17-b272-f5c9b6558552" (UID: "c1c58a9c-aed0-4b17-b272-f5c9b6558552"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:54.220242 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.220216 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-xks-ca" pod "c1c58a9c-aed0-4b17-b272-f5c9b6558552" (UID: "c1c58a9c-aed0-4b17-b272-f5c9b6558552"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:54.220242 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.220236 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c58a9c-aed0-4b17-b272-f5c9b6558552-kube-api-access-8cq9j" pod "c1c58a9c-aed0-4b17-b272-f5c9b6558552" (UID: "c1c58a9c-aed0-4b17-b272-f5c9b6558552"). InnerVolumeSpecName "kube-api-access-8cq9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:54.319788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.319701 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cq9j\" (UniqueName: \"kubernetes.io/projected/c1c58a9c-aed0-4b17-b272-f5c9b6558552-kube-api-access-8cq9j\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:54.319788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.319730 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:54.319788 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:54.319745 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c58a9c-aed0-4b17-b272-f5c9b6558552-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:55.033336 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:55.033294 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" Nov 02 20:31:55.033336 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:55.033328 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j" event={"ID":"c1c58a9c-aed0-4b17-b272-f5c9b6558552","Type":"ContainerDied","Data":"a640d4e150de9fc39b96e14bffac03d9d43d54305b6764f07c2829930d8fdd8e"} Nov 02 20:31:55.033601 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:55.033378 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a640d4e150de9fc39b96e14bffac03d9d43d54305b6764f07c2829930d8fdd8e" Nov 02 20:31:55.743097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:31:55.743064 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:00.144214 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:00.144177 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:03.384045 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:03.384002 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:03.930212 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:03.930165 5850 scope.go:122] "RemoveContainer" containerID="61a167ce4c04d9cd85ade46658415541db4d0fc8cd6fd11e046b366c91073bf9" Nov 02 20:32:03.942359 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:03.942320 5850 scope.go:122] "RemoveContainer" containerID="9512ed6f3b98ea18a6998dd04f43b3f011e6e74e51165c5cbff916ea28d87dfc" Nov 02 20:32:03.952158 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:03.952130 5850 scope.go:122] "RemoveContainer" containerID="bb69ea514cf7dd411e2d437fe059f788309b3dd6b094d63378526a46e601c621" Nov 02 20:32:04.040413 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:04.040362 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:04.567008 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:04.566942 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:09.042344 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:09.042284 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:12.513078 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:12.513036 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:12.924901 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:12.924865 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf"] Nov 02 20:32:12.925373 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:12.925355 5850 cpu_manager.go:414] "containerMap: removing container" podUID="c1c58a9c-aed0-4b17-b272-f5c9b6558552" containerName="revoke-keys" Nov 02 20:32:12.925501 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:12.925375 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c58a9c-aed0-4b17-b272-f5c9b6558552" containerName="revoke-keys" Nov 02 20:32:12.925580 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:12.925509 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c58a9c-aed0-4b17-b272-f5c9b6558552" containerName="revoke-keys" Nov 02 20:32:12.930243 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:12.930220 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:12.984282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:12.984245 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf"] Nov 02 20:32:13.033573 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.033523 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:13.033769 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.033697 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:32:13.108496 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.108446 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.108709 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.108567 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.108709 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.108654 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcrpj\" (UniqueName: \"kubernetes.io/projected/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-kube-api-access-fcrpj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.209522 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.209397 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.209522 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.209447 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcrpj\" (UniqueName: \"kubernetes.io/projected/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-kube-api-access-fcrpj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.209780 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.209603 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.210196 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.210167 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.212065 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.212040 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.267500 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.267441 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcrpj\" (UniqueName: \"kubernetes.io/projected/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-kube-api-access-fcrpj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.444981 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.444948 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:13.543089 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.543006 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:13.911712 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:32:13.911677 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc4ce2e_bf33_4a43_abf2_f48e4faf4f87.slice/crio-7ce624a506fd77120eb048f16d6116e570dfe7dc09f229a80a8f36abb4506c28 WatchSource:0}: Error finding container 7ce624a506fd77120eb048f16d6116e570dfe7dc09f229a80a8f36abb4506c28: Status 404 returned error can't find the container with id 7ce624a506fd77120eb048f16d6116e570dfe7dc09f229a80a8f36abb4506c28 Nov 02 20:32:13.913307 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:13.913269 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf"] Nov 02 20:32:14.131493 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:14.131438 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" event={"ID":"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87","Type":"ContainerStarted","Data":"7ce624a506fd77120eb048f16d6116e570dfe7dc09f229a80a8f36abb4506c28"} Nov 02 20:32:14.760941 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:32:14.760903 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:32:15.138585 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:15.138530 5850 generic.go:356] "Generic (PLEG): container finished" podID="8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87" containerID="f9a00bb2cdf6e9508ed18da529b0c20d0a6efc70f49813075d21e931c6e8d5f9" exitCode=0 Nov 02 20:32:15.138818 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:15.138609 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" event={"ID":"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87","Type":"ContainerDied","Data":"f9a00bb2cdf6e9508ed18da529b0c20d0a6efc70f49813075d21e931c6e8d5f9"} Nov 02 20:32:16.280838 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.280807 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:16.448027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.447935 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-kube-api-access-fcrpj\" (UniqueName: \"kubernetes.io/projected/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-kube-api-access-fcrpj\") pod \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " Nov 02 20:32:16.448027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.447997 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-xks-ca\") pod \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " Nov 02 20:32:16.448282 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.448087 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-openshift-ca\") pod \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\" (UID: \"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87\") " Nov 02 20:32:16.448447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.448418 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-openshift-ca" pod "8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87" (UID: "8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:16.450204 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.450180 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-xks-ca" pod "8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87" (UID: "8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:16.450285 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.450264 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-kube-api-access-fcrpj" pod "8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87" (UID: "8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87"). InnerVolumeSpecName "kube-api-access-fcrpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:16.548989 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.548946 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:16.548989 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.548980 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcrpj\" (UniqueName: \"kubernetes.io/projected/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-kube-api-access-fcrpj\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:16.548989 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:16.548991 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.150564 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:17.150524 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" Nov 02 20:32:17.150564 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:17.150538 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf" event={"ID":"8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87","Type":"ContainerDied","Data":"7ce624a506fd77120eb048f16d6116e570dfe7dc09f229a80a8f36abb4506c28"} Nov 02 20:32:17.150784 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:17.150595 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ce624a506fd77120eb048f16d6116e570dfe7dc09f229a80a8f36abb4506c28" Nov 02 20:32:17.538646 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:17.538548 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:18.234844 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:18.234813 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:22.633066 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:22.633031 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:24.408369 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:24.408335 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:24.467507 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:24.467456 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg"] Nov 02 20:32:24.877090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:24.877059 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:25.039739 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:25.039701 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:26.468905 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:26.468870 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:29.046727 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:29.046686 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48"] Nov 02 20:32:29.097760 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:29.097710 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f118e8d9-prf48"] Nov 02 20:32:29.467963 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:29.467914 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" podUID="5878f8c9-e619-449f-a1df-04ac1b5eb6b9" containerName="payload-processing" containerID="cri-o://d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6" gracePeriod=25 Nov 02 20:32:29.648338 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:29.648298 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" podUID="5878f8c9-e619-449f-a1df-04ac1b5eb6b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:32:29.813346 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:29.813267 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98" path="/var/lib/kubelet/pods/260b6bd5-deaf-4879-9ee7-0cb6a2cfcd98/volumes" Nov 02 20:32:33.155921 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:33.155888 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:33.472827 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:33.472740 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:33.692916 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:33.692876 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf"] Nov 02 20:32:33.744836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:33.744751 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a52dfa-7p2vf"] Nov 02 20:32:33.814389 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:33.814353 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87" path="/var/lib/kubelet/pods/8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87/volumes" Nov 02 20:32:34.149215 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:34.149157 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:39.647945 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:39.647901 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" podUID="5878f8c9-e619-449f-a1df-04ac1b5eb6b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:32:39.953879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:39.953788 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:40.118436 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:40.118395 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:40.268620 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:40.268535 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:44.761488 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:32:44.761432 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:32:49.735795 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:49.735724 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:32:49.897581 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:49.897547 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-plugins-config-volume\") pod \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\" (UID: \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\") " Nov 02 20:32:49.897786 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:49.897610 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-kube-api-access-dqxfr\" (UniqueName: \"kubernetes.io/projected/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-kube-api-access-dqxfr\") pod \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\" (UID: \"5878f8c9-e619-449f-a1df-04ac1b5eb6b9\") " Nov 02 20:32:49.898032 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:49.898005 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-plugins-config-volume" pod "5878f8c9-e619-449f-a1df-04ac1b5eb6b9" (UID: "5878f8c9-e619-449f-a1df-04ac1b5eb6b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:49.899847 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:49.899823 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-kube-api-access-dqxfr" pod "5878f8c9-e619-449f-a1df-04ac1b5eb6b9" (UID: "5878f8c9-e619-449f-a1df-04ac1b5eb6b9"). InnerVolumeSpecName "kube-api-access-dqxfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:49.999509 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:49.999384 5850 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-plugins-config-volume\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:49.999509 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:49.999422 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxfr\" (UniqueName: \"kubernetes.io/projected/5878f8c9-e619-449f-a1df-04ac1b5eb6b9-kube-api-access-dqxfr\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:50.333313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.333227 5850 generic.go:356] "Generic (PLEG): container finished" podID="5878f8c9-e619-449f-a1df-04ac1b5eb6b9" containerID="d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6" exitCode=0 Nov 02 20:32:50.333313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.333270 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" Nov 02 20:32:50.333313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.333300 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" event={"ID":"5878f8c9-e619-449f-a1df-04ac1b5eb6b9","Type":"ContainerDied","Data":"d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6"} Nov 02 20:32:50.333608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.333343 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" event={"ID":"5878f8c9-e619-449f-a1df-04ac1b5eb6b9","Type":"ContainerDied","Data":"23af5c94fd174e900e6d9c284172ef403654e1aa007e05b58968f3ddf8d28353"} Nov 02 20:32:50.333608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.333368 5850 scope.go:122] "RemoveContainer" containerID="d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6" Nov 02 20:32:50.342403 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.342383 5850 scope.go:122] "RemoveContainer" containerID="d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6" Nov 02 20:32:50.342729 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:32:50.342707 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6\": container with ID starting with d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6 not found: ID does not exist" containerID="d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6" Nov 02 20:32:50.342784 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.342739 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6"} err="failed to get container status \"d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6\": rpc error: code = NotFound desc = could not find container \"d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6\": container with ID starting with d365624b00f3a93b5d1c65437bbb530ff27f201ab55295e6a43b4146d272f6f6 not found: ID does not exist" Nov 02 20:32:50.501097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.501063 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg"] Nov 02 20:32:50.552747 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.552713 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg"] Nov 02 20:32:50.648410 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:50.648366 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a52dfa-58f86cb896xhtg" podUID="5878f8c9-e619-449f-a1df-04ac1b5eb6b9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.169:50052\" within 1s: context deadline exceeded" Nov 02 20:32:50.648633 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:32:50.648429 5850 logging.go:55] [core] [Channel #4985 SubChannel #4986]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.169:50052", ServerName: "10.134.0.169:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.169:50052: operation was canceled" Nov 02 20:32:51.814605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:51.814572 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5878f8c9-e619-449f-a1df-04ac1b5eb6b9" path="/var/lib/kubelet/pods/5878f8c9-e619-449f-a1df-04ac1b5eb6b9/volumes" Nov 02 20:32:54.181222 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:54.181183 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:54.532002 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:32:54.531917 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:00.130917 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:33:00.130884 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:09.046350 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:33:09.046306 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n"] Nov 02 20:33:09.097591 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:33:09.097551 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27n"] Nov 02 20:33:09.813371 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:33:09.813335 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e3bb96-c06a-4c9e-b667-a36c18372c26" path="/var/lib/kubelet/pods/66e3bb96-c06a-4c9e-b667-a36c18372c26/volumes" Nov 02 20:33:14.761370 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:33:14.761329 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:33:17.253536 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:33:17.253498 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:24.818414 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:33:24.818375 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:42.083910 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:33:42.083826 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:44.015346 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:33:44.015309 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:44.761958 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:33:44.761921 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:34:03.999609 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:03.999569 5850 scope.go:122] "RemoveContainer" containerID="a6cd1aa79f89e55fc43d46284f1c7e3307f2945c44be0a490f7fc5ded22ef1b0" Nov 02 20:34:04.008337 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:04.008313 5850 scope.go:122] "RemoveContainer" containerID="fef3b215d8461f904d0ea9224cf23249686dbd89349822e28fd53be2e5e0bd6a" Nov 02 20:34:12.759811 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.759771 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf"] Nov 02 20:34:12.760276 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.760172 5850 cpu_manager.go:414] "containerMap: removing container" podUID="5878f8c9-e619-449f-a1df-04ac1b5eb6b9" containerName="payload-processing" Nov 02 20:34:12.760276 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.760184 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5878f8c9-e619-449f-a1df-04ac1b5eb6b9" containerName="payload-processing" Nov 02 20:34:12.760276 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.760196 5850 cpu_manager.go:414] "containerMap: removing container" podUID="8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87" containerName="revoke-keys" Nov 02 20:34:12.760276 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.760201 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87" containerName="revoke-keys" Nov 02 20:34:12.760527 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.760286 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="5878f8c9-e619-449f-a1df-04ac1b5eb6b9" containerName="payload-processing" Nov 02 20:34:12.760527 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.760294 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc4ce2e-bf33-4a43-abf2-f48e4faf4f87" containerName="revoke-keys" Nov 02 20:34:12.763451 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.763433 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:12.822047 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.822008 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf"] Nov 02 20:34:12.871429 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.871392 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:34:12.871627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.871466 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:34:12.925858 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.925821 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rfcrf\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:12.926074 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.925971 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rfcrf\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:12.926162 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:12.926135 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsnj6\" (UniqueName: \"kubernetes.io/projected/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-kube-api-access-wsnj6\") pod \"maas-api-revoke-sub-452b6d7b-rfcrf\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:13.027447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:13.027355 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rfcrf\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:13.027447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:13.027424 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rfcrf\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:13.027677 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:13.027485 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsnj6\" (UniqueName: \"kubernetes.io/projected/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-kube-api-access-wsnj6\") pod \"maas-api-revoke-sub-452b6d7b-rfcrf\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:13.028205 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:13.028184 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rfcrf\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:13.029808 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:13.029785 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rfcrf\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:13.086181 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:13.086137 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsnj6\" (UniqueName: \"kubernetes.io/projected/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-kube-api-access-wsnj6\") pod \"maas-api-revoke-sub-452b6d7b-rfcrf\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:13.375939 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:13.375901 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:13.745684 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:34:13.745616 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11855b7_41e7_4592_9f61_ccbbdc5b3df8.slice/crio-b1515b406825e1bb613fd1453e2b801742d793a87dfc6b669b02bf3ed0ed590a WatchSource:0}: Error finding container b1515b406825e1bb613fd1453e2b801742d793a87dfc6b669b02bf3ed0ed590a: Status 404 returned error can't find the container with id b1515b406825e1bb613fd1453e2b801742d793a87dfc6b669b02bf3ed0ed590a Nov 02 20:34:13.746977 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:13.746950 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf"] Nov 02 20:34:13.814205 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:13.814171 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" event={"ID":"e11855b7-41e7-4592-9f61-ccbbdc5b3df8","Type":"ContainerStarted","Data":"b1515b406825e1bb613fd1453e2b801742d793a87dfc6b669b02bf3ed0ed590a"} Nov 02 20:34:14.776603 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:34:14.776556 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:34:14.823077 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:14.823037 5850 generic.go:356] "Generic (PLEG): container finished" podID="e11855b7-41e7-4592-9f61-ccbbdc5b3df8" containerID="a290e8aa411a60d3cb26183a7ad0dc662b05edc7923f245e6783ba09998c7742" exitCode=0 Nov 02 20:34:14.823639 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:14.823120 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" event={"ID":"e11855b7-41e7-4592-9f61-ccbbdc5b3df8","Type":"ContainerDied","Data":"a290e8aa411a60d3cb26183a7ad0dc662b05edc7923f245e6783ba09998c7742"} Nov 02 20:34:16.711506 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.710774 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:16.796824 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.796778 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-openshift-ca\") pod \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " Nov 02 20:34:16.797741 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.797712 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-kube-api-access-wsnj6\" (UniqueName: \"kubernetes.io/projected/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-kube-api-access-wsnj6\") pod \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " Nov 02 20:34:16.798330 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.798309 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-xks-ca\") pod \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\" (UID: \"e11855b7-41e7-4592-9f61-ccbbdc5b3df8\") " Nov 02 20:34:16.799147 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.797468 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-openshift-ca" pod "e11855b7-41e7-4592-9f61-ccbbdc5b3df8" (UID: "e11855b7-41e7-4592-9f61-ccbbdc5b3df8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:34:16.800734 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.800674 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-kube-api-access-wsnj6" pod "e11855b7-41e7-4592-9f61-ccbbdc5b3df8" (UID: "e11855b7-41e7-4592-9f61-ccbbdc5b3df8"). InnerVolumeSpecName "kube-api-access-wsnj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:16.803398 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.803357 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-xks-ca" pod "e11855b7-41e7-4592-9f61-ccbbdc5b3df8" (UID: "e11855b7-41e7-4592-9f61-ccbbdc5b3df8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:34:16.976700 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.976655 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsnj6\" (UniqueName: \"kubernetes.io/projected/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-kube-api-access-wsnj6\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:16.990088 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.990045 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:16.993729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:16.993698 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11855b7-41e7-4592-9f61-ccbbdc5b3df8-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:17.060671 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:17.060559 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" event={"ID":"e11855b7-41e7-4592-9f61-ccbbdc5b3df8","Type":"ContainerDied","Data":"b1515b406825e1bb613fd1453e2b801742d793a87dfc6b669b02bf3ed0ed590a"} Nov 02 20:34:17.060974 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:17.060952 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1515b406825e1bb613fd1453e2b801742d793a87dfc6b669b02bf3ed0ed590a" Nov 02 20:34:17.061195 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:17.061149 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf" Nov 02 20:34:25.205409 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:25.205371 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:25.365101 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:25.365065 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:27.176090 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:27.176053 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:27.869637 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:27.869601 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:33.136245 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.136206 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n"] Nov 02 20:34:33.136669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.136638 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e11855b7-41e7-4592-9f61-ccbbdc5b3df8" containerName="revoke-keys" Nov 02 20:34:33.136669 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.136651 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11855b7-41e7-4592-9f61-ccbbdc5b3df8" containerName="revoke-keys" Nov 02 20:34:33.136743 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.136726 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11855b7-41e7-4592-9f61-ccbbdc5b3df8" containerName="revoke-keys" Nov 02 20:34:33.140042 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.140016 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.194851 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.194780 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:34:33.196428 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.196397 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n"] Nov 02 20:34:33.204488 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.204440 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5hkd\" (UniqueName: \"kubernetes.io/projected/e91c0ae0-e400-4479-8919-a92261bbd443-kube-api-access-v5hkd\") pod \"maas-api-revoke-sub-3e31a506-45v2n\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.204657 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.204524 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-45v2n\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.204657 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.204556 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-45v2n\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.245729 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.245697 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:34:33.305858 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.305812 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5hkd\" (UniqueName: \"kubernetes.io/projected/e91c0ae0-e400-4479-8919-a92261bbd443-kube-api-access-v5hkd\") pod \"maas-api-revoke-sub-3e31a506-45v2n\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.306050 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.305871 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-45v2n\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.306050 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.305910 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-45v2n\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.306767 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.306740 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-45v2n\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.308419 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.308393 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-45v2n\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.364606 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.364566 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5hkd\" (UniqueName: \"kubernetes.io/projected/e91c0ae0-e400-4479-8919-a92261bbd443-kube-api-access-v5hkd\") pod \"maas-api-revoke-sub-3e31a506-45v2n\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.453325 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.453225 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:33.819827 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:34:33.819795 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91c0ae0_e400_4479_8919_a92261bbd443.slice/crio-70eae92992ce5748b94c363e731b17030ea8d598ae1d2aebacc8a6a0e98af84a WatchSource:0}: Error finding container 70eae92992ce5748b94c363e731b17030ea8d598ae1d2aebacc8a6a0e98af84a: Status 404 returned error can't find the container with id 70eae92992ce5748b94c363e731b17030ea8d598ae1d2aebacc8a6a0e98af84a Nov 02 20:34:33.821634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:33.821607 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n"] Nov 02 20:34:34.166663 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:34.166621 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" event={"ID":"e91c0ae0-e400-4479-8919-a92261bbd443","Type":"ContainerStarted","Data":"cb2186d06395d04bf79b7d05ae2e3de3914b6815636e5f5e894c5bfcbe759781"} Nov 02 20:34:34.167190 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:34.166695 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" event={"ID":"e91c0ae0-e400-4479-8919-a92261bbd443","Type":"ContainerStarted","Data":"70eae92992ce5748b94c363e731b17030ea8d598ae1d2aebacc8a6a0e98af84a"} Nov 02 20:34:34.282804 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:34.282746 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" podStartSLOduration=1.2827309470000001 podStartE2EDuration="1.282730947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:34.282112632 +0000 UTC m=+3932.624501713" watchObservedRunningTime="2026-11-02 20:34:34.282730947 +0000 UTC m=+3932.625120026" Nov 02 20:34:35.173575 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:35.173530 5850 generic.go:356] "Generic (PLEG): container finished" podID="e91c0ae0-e400-4479-8919-a92261bbd443" containerID="cb2186d06395d04bf79b7d05ae2e3de3914b6815636e5f5e894c5bfcbe759781" exitCode=0 Nov 02 20:34:35.173997 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:35.173622 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" event={"ID":"e91c0ae0-e400-4479-8919-a92261bbd443","Type":"ContainerDied","Data":"cb2186d06395d04bf79b7d05ae2e3de3914b6815636e5f5e894c5bfcbe759781"} Nov 02 20:34:36.387414 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.387376 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:36.440016 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.439905 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-openshift-ca\") pod \"e91c0ae0-e400-4479-8919-a92261bbd443\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " Nov 02 20:34:36.440213 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.440043 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-xks-ca\") pod \"e91c0ae0-e400-4479-8919-a92261bbd443\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " Nov 02 20:34:36.440213 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.440139 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91c0ae0-e400-4479-8919-a92261bbd443-kube-api-access-v5hkd\" (UniqueName: \"kubernetes.io/projected/e91c0ae0-e400-4479-8919-a92261bbd443-kube-api-access-v5hkd\") pod \"e91c0ae0-e400-4479-8919-a92261bbd443\" (UID: \"e91c0ae0-e400-4479-8919-a92261bbd443\") " Nov 02 20:34:36.440790 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.440754 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-openshift-ca" pod "e91c0ae0-e400-4479-8919-a92261bbd443" (UID: "e91c0ae0-e400-4479-8919-a92261bbd443"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:34:36.443228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.443193 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91c0ae0-e400-4479-8919-a92261bbd443-kube-api-access-v5hkd" pod "e91c0ae0-e400-4479-8919-a92261bbd443" (UID: "e91c0ae0-e400-4479-8919-a92261bbd443"). InnerVolumeSpecName "kube-api-access-v5hkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:36.443228 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.443203 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-xks-ca" pod "e91c0ae0-e400-4479-8919-a92261bbd443" (UID: "e91c0ae0-e400-4479-8919-a92261bbd443"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:34:36.542039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.541994 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:36.542039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.542039 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91c0ae0-e400-4479-8919-a92261bbd443-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:36.542318 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:36.542054 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5hkd\" (UniqueName: \"kubernetes.io/projected/e91c0ae0-e400-4479-8919-a92261bbd443-kube-api-access-v5hkd\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:37.213499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:37.209678 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" event={"ID":"e91c0ae0-e400-4479-8919-a92261bbd443","Type":"ContainerDied","Data":"70eae92992ce5748b94c363e731b17030ea8d598ae1d2aebacc8a6a0e98af84a"} Nov 02 20:34:37.213499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:37.209749 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70eae92992ce5748b94c363e731b17030ea8d598ae1d2aebacc8a6a0e98af84a" Nov 02 20:34:37.213499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:37.209824 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n" Nov 02 20:34:42.727551 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:42.727512 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:42.888879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:34:42.888850 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:44.761352 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:34:44.761304 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:35:00.287741 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.287706 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh"] Nov 02 20:35:00.288171 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.288117 5850 cpu_manager.go:414] "containerMap: removing container" podUID="e91c0ae0-e400-4479-8919-a92261bbd443" containerName="revoke-keys" Nov 02 20:35:00.288171 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.288129 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e91c0ae0-e400-4479-8919-a92261bbd443" containerName="revoke-keys" Nov 02 20:35:00.288260 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.288211 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="e91c0ae0-e400-4479-8919-a92261bbd443" containerName="revoke-keys" Nov 02 20:35:00.291867 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.291849 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.396913 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.396882 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:35:00.397133 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.397066 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:35:00.398730 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.398710 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh"] Nov 02 20:35:00.405526 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.405452 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6476e994-qnrlh\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.405526 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.405517 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275x4\" (UniqueName: \"kubernetes.io/projected/ff7c5573-c7ee-4876-984c-7f034ee569fd-kube-api-access-275x4\") pod \"maas-api-revoke-sub-6476e994-qnrlh\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.405782 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.405751 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6476e994-qnrlh\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.506574 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.506529 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6476e994-qnrlh\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.506789 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.506620 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6476e994-qnrlh\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.506789 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.506659 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275x4\" (UniqueName: \"kubernetes.io/projected/ff7c5573-c7ee-4876-984c-7f034ee569fd-kube-api-access-275x4\") pod \"maas-api-revoke-sub-6476e994-qnrlh\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.507246 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.507225 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6476e994-qnrlh\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.509109 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.509084 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6476e994-qnrlh\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.565316 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.565242 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275x4\" (UniqueName: \"kubernetes.io/projected/ff7c5573-c7ee-4876-984c-7f034ee569fd-kube-api-access-275x4\") pod \"maas-api-revoke-sub-6476e994-qnrlh\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.604970 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.604928 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:00.973500 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:35:00.973449 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7c5573_c7ee_4876_984c_7f034ee569fd.slice/crio-6326042612e95f54b16396f8bc9a9a95a6e8fba0ba4777935997ba7702514d6e WatchSource:0}: Error finding container 6326042612e95f54b16396f8bc9a9a95a6e8fba0ba4777935997ba7702514d6e: Status 404 returned error can't find the container with id 6326042612e95f54b16396f8bc9a9a95a6e8fba0ba4777935997ba7702514d6e Nov 02 20:35:00.974835 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:00.974810 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh"] Nov 02 20:35:01.047850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:01.047814 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64"] Nov 02 20:35:01.052352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:01.052321 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-lvb64"] Nov 02 20:35:01.419104 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:01.419069 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" event={"ID":"ff7c5573-c7ee-4876-984c-7f034ee569fd","Type":"ContainerStarted","Data":"2cdcf2fd1458a8524496ef1d5f39378180334bdf36b79523a8cb0022ef724ee3"} Nov 02 20:35:01.419561 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:01.419123 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" event={"ID":"ff7c5573-c7ee-4876-984c-7f034ee569fd","Type":"ContainerStarted","Data":"6326042612e95f54b16396f8bc9a9a95a6e8fba0ba4777935997ba7702514d6e"} Nov 02 20:35:01.534946 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:01.534895 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" podStartSLOduration=1.534880363 podStartE2EDuration="1.534880363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:01.534421059 +0000 UTC m=+3959.876810139" watchObservedRunningTime="2026-11-02 20:35:01.534880363 +0000 UTC m=+3959.877269442" Nov 02 20:35:01.813798 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:01.813767 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb3107e-83c7-483b-b696-e0b6e8fac49d" path="/var/lib/kubelet/pods/7eb3107e-83c7-483b-b696-e0b6e8fac49d/volumes" Nov 02 20:35:02.426010 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:02.425974 5850 generic.go:356] "Generic (PLEG): container finished" podID="ff7c5573-c7ee-4876-984c-7f034ee569fd" containerID="2cdcf2fd1458a8524496ef1d5f39378180334bdf36b79523a8cb0022ef724ee3" exitCode=0 Nov 02 20:35:02.426435 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:02.426033 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" event={"ID":"ff7c5573-c7ee-4876-984c-7f034ee569fd","Type":"ContainerDied","Data":"2cdcf2fd1458a8524496ef1d5f39378180334bdf36b79523a8cb0022ef724ee3"} Nov 02 20:35:03.561347 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.561316 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:03.636362 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.636327 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7c5573-c7ee-4876-984c-7f034ee569fd-kube-api-access-275x4\" (UniqueName: \"kubernetes.io/projected/ff7c5573-c7ee-4876-984c-7f034ee569fd-kube-api-access-275x4\") pod \"ff7c5573-c7ee-4876-984c-7f034ee569fd\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " Nov 02 20:35:03.636572 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.636421 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-openshift-ca\") pod \"ff7c5573-c7ee-4876-984c-7f034ee569fd\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " Nov 02 20:35:03.636572 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.636522 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-xks-ca\") pod \"ff7c5573-c7ee-4876-984c-7f034ee569fd\" (UID: \"ff7c5573-c7ee-4876-984c-7f034ee569fd\") " Nov 02 20:35:03.636836 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.636812 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-openshift-ca" pod "ff7c5573-c7ee-4876-984c-7f034ee569fd" (UID: "ff7c5573-c7ee-4876-984c-7f034ee569fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:03.638588 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.638563 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-xks-ca" pod "ff7c5573-c7ee-4876-984c-7f034ee569fd" (UID: "ff7c5573-c7ee-4876-984c-7f034ee569fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:03.638683 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.638646 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7c5573-c7ee-4876-984c-7f034ee569fd-kube-api-access-275x4" pod "ff7c5573-c7ee-4876-984c-7f034ee569fd" (UID: "ff7c5573-c7ee-4876-984c-7f034ee569fd"). InnerVolumeSpecName "kube-api-access-275x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:03.737990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.737895 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-xks-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:03.737990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.737933 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275x4\" (UniqueName: \"kubernetes.io/projected/ff7c5573-c7ee-4876-984c-7f034ee569fd-kube-api-access-275x4\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:03.737990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:03.737943 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7c5573-c7ee-4876-984c-7f034ee569fd-maas-api-openshift-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:04.033145 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:04.033059 5850 scope.go:122] "RemoveContainer" containerID="ee08e0824b9270f575980917f51daced2df9716f10cdd505505ae4668a0bbedd" Nov 02 20:35:04.437135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:04.437106 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" Nov 02 20:35:04.437135 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:04.437126 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh" event={"ID":"ff7c5573-c7ee-4876-984c-7f034ee569fd","Type":"ContainerDied","Data":"6326042612e95f54b16396f8bc9a9a95a6e8fba0ba4777935997ba7702514d6e"} Nov 02 20:35:04.437352 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:04.437174 5850 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6326042612e95f54b16396f8bc9a9a95a6e8fba0ba4777935997ba7702514d6e" Nov 02 20:35:13.338891 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.338790 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k"] Nov 02 20:35:13.339448 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.339428 5850 cpu_manager.go:414] "containerMap: removing container" podUID="ff7c5573-c7ee-4876-984c-7f034ee569fd" containerName="revoke-keys" Nov 02 20:35:13.339550 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.339452 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7c5573-c7ee-4876-984c-7f034ee569fd" containerName="revoke-keys" Nov 02 20:35:13.339623 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.339615 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7c5573-c7ee-4876-984c-7f034ee569fd" containerName="revoke-keys" Nov 02 20:35:13.343861 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.343837 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.398288 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.398252 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k"] Nov 02 20:35:13.435120 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.435081 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqlwp\" (UniqueName: \"kubernetes.io/projected/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-kube-api-access-kqlwp\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.435334 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.435140 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-maas-api-tls\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.435334 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.435251 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-openshift-service-ca\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.435334 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.435305 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-metrics-certs\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.447753 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.447678 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 20:35:13.447753 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.447741 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 20:35:13.447986 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.447787 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sg6rf\"" Nov 02 20:35:13.536632 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.536591 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-maas-api-tls\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.536837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.536643 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-openshift-service-ca\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.536837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.536677 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-metrics-certs\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.536837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.536775 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqlwp\" (UniqueName: \"kubernetes.io/projected/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-kube-api-access-kqlwp\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.537388 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.537363 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-openshift-service-ca\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.539137 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.539113 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-maas-api-tls\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.539298 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.539277 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-metrics-certs\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.596067 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.595994 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqlwp\" (UniqueName: \"kubernetes.io/projected/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-kube-api-access-kqlwp\") pod \"maas-api-769845dd8d-qkk8k\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:13.660322 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:13.660282 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:14.032692 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:35:14.032660 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a435e7_b92e_4e5f_bc00_4aa007e47cfd.slice/crio-022480e844040475fcc31cf67e0ceb924ee5df4724536d83ae762c6827d3fccc WatchSource:0}: Error finding container 022480e844040475fcc31cf67e0ceb924ee5df4724536d83ae762c6827d3fccc: Status 404 returned error can't find the container with id 022480e844040475fcc31cf67e0ceb924ee5df4724536d83ae762c6827d3fccc Nov 02 20:35:14.034243 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:14.034221 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k"] Nov 02 20:35:14.034433 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:14.034417 5850 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:35:14.488460 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:14.488424 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" event={"ID":"35a435e7-b92e-4e5f-bc00-4aa007e47cfd","Type":"ContainerStarted","Data":"022480e844040475fcc31cf67e0ceb924ee5df4724536d83ae762c6827d3fccc"} Nov 02 20:35:14.760600 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:35:14.760513 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:35:15.496211 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:15.496162 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" event={"ID":"35a435e7-b92e-4e5f-bc00-4aa007e47cfd","Type":"ContainerStarted","Data":"938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d"} Nov 02 20:35:15.496683 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:15.496343 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:15.613608 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:15.613539 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" podStartSLOduration=1.8196324640000001 podStartE2EDuration="2.613520662s" totalImagesPullingTime="793.888198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:15.613096839 +0000 UTC m=+3973.955485943" watchObservedRunningTime="2026-11-02 20:35:15.613520662 +0000 UTC m=+3973.955909743" Nov 02 20:35:19.800605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:19.800555 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:20.339816 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.339780 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26"] Nov 02 20:35:20.343793 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.343768 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.437884 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.437855 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5b04055d-openshift-default-dockercfg-bgqq7\"" Nov 02 20:35:20.442008 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.441981 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:20.451534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.451503 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26"] Nov 02 20:35:20.511287 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.511240 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-podinfo\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.511534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.511290 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-envoy\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.511534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.511390 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwjs8\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-kube-api-access-lwjs8\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.511534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.511419 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-credential-socket\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.511534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.511459 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8a7756-9799-4da8-8afe-39ef1635de72-istiod-ca-cert\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.511534 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.511506 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-certs\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.511787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.511582 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-token\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.511787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.511631 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-socket\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.511787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.511696 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-data\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.599739 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.599633 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:20.612270 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612236 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8a7756-9799-4da8-8afe-39ef1635de72-istiod-ca-cert\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.612270 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612276 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-certs\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.612556 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612297 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-token\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.612556 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612415 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-socket\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.612818 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612627 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-data\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.612818 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612666 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-podinfo\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.612818 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612732 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-envoy\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.612818 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612780 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-certs\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.612818 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612805 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwjs8\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-kube-api-access-lwjs8\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.613141 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612842 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-socket\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.613141 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612869 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-credential-socket\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.613141 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.612999 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-data\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.613279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.613171 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8a7756-9799-4da8-8afe-39ef1635de72-istiod-ca-cert\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.613629 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.613610 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-credential-socket\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.615108 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.615084 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-envoy\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.615939 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.615917 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-podinfo\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.670605 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.670573 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-token\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.670965 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.670942 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwjs8\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-kube-api-access-lwjs8\") pod \"e2e-ait-5b04055d-openshift-default-7795fd879-pgd26\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:20.767150 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.767116 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:20.957373 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:20.957333 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:21.379796 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:21.379756 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26"] Nov 02 20:35:21.415661 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:35:21.415631 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8a7756_9799_4da8_8afe_39ef1635de72.slice/crio-9d55639b0d0899ba6d0c7b64c40bc3d2eaa405f0828acc4f7f62662ef54b3359 WatchSource:0}: Error finding container 9d55639b0d0899ba6d0c7b64c40bc3d2eaa405f0828acc4f7f62662ef54b3359: Status 404 returned error can't find the container with id 9d55639b0d0899ba6d0c7b64c40bc3d2eaa405f0828acc4f7f62662ef54b3359 Nov 02 20:35:21.417856 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:21.417824 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:35:21.417942 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:21.417909 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:35:21.417942 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:21.417938 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:35:21.506878 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:21.506844 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:21.527935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:21.527898 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" event={"ID":"9a8a7756-9799-4da8-8afe-39ef1635de72","Type":"ContainerStarted","Data":"0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35"} Nov 02 20:35:21.528127 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:21.527965 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" event={"ID":"9a8a7756-9799-4da8-8afe-39ef1635de72","Type":"ContainerStarted","Data":"9d55639b0d0899ba6d0c7b64c40bc3d2eaa405f0828acc4f7f62662ef54b3359"} Nov 02 20:35:21.746074 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:21.745955 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" podStartSLOduration=1.745935901 podStartE2EDuration="1.745935901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:21.743757914 +0000 UTC m=+3980.086146994" watchObservedRunningTime="2026-11-02 20:35:21.745935901 +0000 UTC m=+3980.088324984" Nov 02 20:35:21.957607 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:21.957576 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:22.054821 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.054736 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f6584b448-px4lz"] Nov 02 20:35:22.058441 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.058418 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.115422 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.115391 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f6584b448-px4lz"] Nov 02 20:35:22.234825 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.234783 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctvp\" (UniqueName: \"kubernetes.io/projected/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-kube-api-access-bctvp\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.234825 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.234831 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-maas-api-tls\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.235055 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.234986 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-metrics-certs\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.235055 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.235032 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-openshift-service-ca\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.335998 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.335898 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctvp\" (UniqueName: \"kubernetes.io/projected/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-kube-api-access-bctvp\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.335998 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.335945 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-maas-api-tls\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.335998 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.336002 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-metrics-certs\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.336334 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.336026 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-openshift-service-ca\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.336693 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.336671 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-openshift-service-ca\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.338750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.338727 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-maas-api-tls\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.338833 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.338785 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-metrics-certs\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.394241 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.394205 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctvp\" (UniqueName: \"kubernetes.io/projected/5cea4586-2f7f-46cc-a090-dd11d4b0f7d5-kube-api-access-bctvp\") pod \"maas-api-f6584b448-px4lz\" (UID: \"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5\") " pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.672397 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.672352 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:22.962880 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:22.962794 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:23.042150 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:35:23.042115 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cea4586_2f7f_46cc_a090_dd11d4b0f7d5.slice/crio-1939581521d9c53945452ced0326069188a067ff4c2a0eb40159fea5c336a5b6 WatchSource:0}: Error finding container 1939581521d9c53945452ced0326069188a067ff4c2a0eb40159fea5c336a5b6: Status 404 returned error can't find the container with id 1939581521d9c53945452ced0326069188a067ff4c2a0eb40159fea5c336a5b6 Nov 02 20:35:23.069850 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:23.069813 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f6584b448-px4lz"] Nov 02 20:35:23.538715 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:23.538678 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" event={"ID":"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5","Type":"ContainerStarted","Data":"1939581521d9c53945452ced0326069188a067ff4c2a0eb40159fea5c336a5b6"} Nov 02 20:35:23.538961 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:23.538942 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:23.539931 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:23.539913 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:23.804893 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:23.804806 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:24.030831 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:24.030792 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:24.199687 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:24.199658 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:24.544823 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:24.544726 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" event={"ID":"5cea4586-2f7f-46cc-a090-dd11d4b0f7d5","Type":"ContainerStarted","Data":"ff418954c36a701d491bfd545b5ad73aef9d3c2e5792cfe6581b0f4269f021ac"} Nov 02 20:35:24.544996 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:24.544888 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:24.663034 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:24.662980 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" podStartSLOduration=1.8030327659999998 podStartE2EDuration="2.662967126s" totalImagesPullingTime="859.93436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:24.662267634 +0000 UTC m=+3983.004656716" watchObservedRunningTime="2026-11-02 20:35:24.662967126 +0000 UTC m=+3983.005356263" Nov 02 20:35:25.556750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:25.556714 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:25.926520 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:25.926486 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:26.518943 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:26.518905 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:27.042391 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:27.042340 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:27.214996 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:27.214946 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:27.368971 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:27.368942 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:27.560256 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:27.559159 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:29.091031 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:29.090991 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q"] Nov 02 20:35:29.142645 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:29.142604 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ps52q"] Nov 02 20:35:29.813747 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:29.813714 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9243df2e-1a33-4f3c-99c0-09eab9fbcd80" path="/var/lib/kubelet/pods/9243df2e-1a33-4f3c-99c0-09eab9fbcd80/volumes" Nov 02 20:35:30.520837 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:30.520798 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:30.555817 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:30.555776 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f6584b448-px4lz" Nov 02 20:35:30.727447 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:30.727414 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k"] Nov 02 20:35:30.727730 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:30.727681 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" podUID="35a435e7-b92e-4e5f-bc00-4aa007e47cfd" containerName="maas-api" containerID="cri-o://938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d" gracePeriod=30 Nov 02 20:35:30.993340 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:30.993307 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:31.032615 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.032572 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-openshift-service-ca\") pod \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " Nov 02 20:35:31.032807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.032686 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-metrics-certs\") pod \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " Nov 02 20:35:31.032807 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.032728 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-maas-api-tls\") pod \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " Nov 02 20:35:31.032903 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.032837 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-kube-api-access-kqlwp\" (UniqueName: \"kubernetes.io/projected/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-kube-api-access-kqlwp\") pod \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\" (UID: \"35a435e7-b92e-4e5f-bc00-4aa007e47cfd\") " Nov 02 20:35:31.033024 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.032996 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-openshift-service-ca" pod "35a435e7-b92e-4e5f-bc00-4aa007e47cfd" (UID: "35a435e7-b92e-4e5f-bc00-4aa007e47cfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:31.033222 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.033198 5850 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-openshift-service-ca\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:31.034961 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.034935 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-metrics-certs" pod "35a435e7-b92e-4e5f-bc00-4aa007e47cfd" (UID: "35a435e7-b92e-4e5f-bc00-4aa007e47cfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:31.035075 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.035008 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-maas-api-tls" pod "35a435e7-b92e-4e5f-bc00-4aa007e47cfd" (UID: "35a435e7-b92e-4e5f-bc00-4aa007e47cfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:31.035142 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.035127 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-kube-api-access-kqlwp" pod "35a435e7-b92e-4e5f-bc00-4aa007e47cfd" (UID: "35a435e7-b92e-4e5f-bc00-4aa007e47cfd"). InnerVolumeSpecName "kube-api-access-kqlwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:31.134694 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.134652 5850 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-metrics-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:31.134694 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.134689 5850 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-maas-api-tls\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:31.134935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.134706 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqlwp\" (UniqueName: \"kubernetes.io/projected/35a435e7-b92e-4e5f-bc00-4aa007e47cfd-kube-api-access-kqlwp\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:31.597196 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.597110 5850 generic.go:356] "Generic (PLEG): container finished" podID="35a435e7-b92e-4e5f-bc00-4aa007e47cfd" containerID="938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d" exitCode=0 Nov 02 20:35:31.597196 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.597167 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" Nov 02 20:35:31.597731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.597193 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" event={"ID":"35a435e7-b92e-4e5f-bc00-4aa007e47cfd","Type":"ContainerDied","Data":"938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d"} Nov 02 20:35:31.597731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.597266 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k" event={"ID":"35a435e7-b92e-4e5f-bc00-4aa007e47cfd","Type":"ContainerDied","Data":"022480e844040475fcc31cf67e0ceb924ee5df4724536d83ae762c6827d3fccc"} Nov 02 20:35:31.597731 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.597289 5850 scope.go:122] "RemoveContainer" containerID="938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d" Nov 02 20:35:31.609410 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.609046 5850 scope.go:122] "RemoveContainer" containerID="938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d" Nov 02 20:35:31.610362 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:35:31.610336 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d\": container with ID starting with 938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d not found: ID does not exist" containerID="938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d" Nov 02 20:35:31.610499 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.610368 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d"} err="failed to get container status \"938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d\": rpc error: code = NotFound desc = could not find container \"938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d\": container with ID starting with 938c3cdf8912d49defd66eb07cf6f30b1a3857eee8f87551b6722f7ada2fd47d not found: ID does not exist" Nov 02 20:35:31.764900 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.764862 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k"] Nov 02 20:35:31.767968 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.767941 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-769845dd8d-qkk8k"] Nov 02 20:35:31.813646 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:31.813611 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a435e7-b92e-4e5f-bc00-4aa007e47cfd" path="/var/lib/kubelet/pods/35a435e7-b92e-4e5f-bc00-4aa007e47cfd/volumes" Nov 02 20:35:32.032033 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:32.031996 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:35.551846 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:35.551814 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:36.445593 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:36.445553 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:40.847624 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:40.847576 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:44.681442 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:44.681402 5850 generic.go:356] "Generic (PLEG): container finished" podID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerID="0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35" exitCode=0 Nov 02 20:35:44.681980 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:44.681497 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" event={"ID":"9a8a7756-9799-4da8-8afe-39ef1635de72","Type":"ContainerDied","Data":"0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35"} Nov 02 20:35:44.681980 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:44.681912 5850 scope.go:122] "RemoveContainer" containerID="0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35" Nov 02 20:35:44.683230 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:44.683194 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:35:44.683357 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:44.683287 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:35:44.683357 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:44.683337 5850 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:35:44.763763 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:35:44.763717 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:35:45.580491 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:45.580451 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:45.688541 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:45.688503 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" event={"ID":"9a8a7756-9799-4da8-8afe-39ef1635de72","Type":"ContainerStarted","Data":"aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9"} Nov 02 20:35:45.958288 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:45.958255 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:45.958533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:45.958428 5850 patch_prober.go:29] interesting pod/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 20:35:45.958533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:45.958498 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 20:35:46.957982 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:46.957940 5850 patch_prober.go:29] interesting pod/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 20:35:46.958539 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:46.958014 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 20:35:47.957867 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:47.957834 5850 patch_prober.go:29] interesting pod/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 20:35:47.958067 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:47.957886 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 20:35:48.958769 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:48.958728 5850 patch_prober.go:29] interesting pod/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 20:35:48.959196 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:48.958794 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 20:35:49.957674 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:49.957638 5850 patch_prober.go:29] interesting pod/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 20:35:49.957864 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:49.957691 5850 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 20:35:50.036766 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:50.036727 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:50.962102 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:50.962071 5850 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:50.962379 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:50.962337 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:50.963097 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:50.963080 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:35:54.436121 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:54.436074 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:58.860043 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:35:58.859994 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:03.243044 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:03.242997 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:04.057351 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:04.057309 5850 scope.go:122] "RemoveContainer" containerID="b87cc40cb9066dc7fd28028c2484c4b793143062e14fb3bc76aeee3750f283c0" Nov 02 20:36:07.242726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:07.242695 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:07.399923 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:07.399862 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:07.630603 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:07.630572 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:11.488843 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:11.488812 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:12.925779 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:12.925749 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:13.175148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:13.175109 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:14.761100 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:36:14.761060 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:36:16.183552 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:16.183515 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26"] Nov 02 20:36:16.184049 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:16.183781 5850 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" containerID="cri-o://aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9" gracePeriod=30 Nov 02 20:36:16.210667 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:16.210626 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:17.401798 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:17.401758 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:17.975106 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:17.975074 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:18.470222 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:18.470184 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:20.964994 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:20.964949 5850 patch_prober.go:29] interesting pod/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:36:20.965403 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:20.965029 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:36:21.445754 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.445722 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:36:21.528327 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528292 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-certs\") pod \"9a8a7756-9799-4da8-8afe-39ef1635de72\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " Nov 02 20:36:21.528327 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528342 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-token\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-token\") pod \"9a8a7756-9799-4da8-8afe-39ef1635de72\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " Nov 02 20:36:21.528626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528364 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8a7756-9799-4da8-8afe-39ef1635de72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8a7756-9799-4da8-8afe-39ef1635de72-istiod-ca-cert\") pod \"9a8a7756-9799-4da8-8afe-39ef1635de72\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " Nov 02 20:36:21.528626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528387 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-socket\") pod \"9a8a7756-9799-4da8-8afe-39ef1635de72\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " Nov 02 20:36:21.528626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528413 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-data\") pod \"9a8a7756-9799-4da8-8afe-39ef1635de72\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " Nov 02 20:36:21.528626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528459 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-podinfo\") pod \"9a8a7756-9799-4da8-8afe-39ef1635de72\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " Nov 02 20:36:21.528626 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528583 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-kube-api-access-lwjs8\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-kube-api-access-lwjs8\") pod \"9a8a7756-9799-4da8-8afe-39ef1635de72\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " Nov 02 20:36:21.528898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528666 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-envoy\") pod \"9a8a7756-9799-4da8-8afe-39ef1635de72\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " Nov 02 20:36:21.528898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528726 5850 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-credential-socket\") pod \"9a8a7756-9799-4da8-8afe-39ef1635de72\" (UID: \"9a8a7756-9799-4da8-8afe-39ef1635de72\") " Nov 02 20:36:21.528898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528739 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-certs" pod "9a8a7756-9799-4da8-8afe-39ef1635de72" (UID: "9a8a7756-9799-4da8-8afe-39ef1635de72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:21.528898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528770 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-socket" pod "9a8a7756-9799-4da8-8afe-39ef1635de72" (UID: "9a8a7756-9799-4da8-8afe-39ef1635de72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:21.528898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528781 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-data" pod "9a8a7756-9799-4da8-8afe-39ef1635de72" (UID: "9a8a7756-9799-4da8-8afe-39ef1635de72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:21.528898 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.528860 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8a7756-9799-4da8-8afe-39ef1635de72-istiod-ca-cert" pod "9a8a7756-9799-4da8-8afe-39ef1635de72" (UID: "9a8a7756-9799-4da8-8afe-39ef1635de72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:21.529238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.529080 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-credential-socket" pod "9a8a7756-9799-4da8-8afe-39ef1635de72" (UID: "9a8a7756-9799-4da8-8afe-39ef1635de72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:21.529238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.529102 5850 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-certs\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:21.529238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.529116 5850 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8a7756-9799-4da8-8afe-39ef1635de72-istiod-ca-cert\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:21.529238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.529126 5850 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-workload-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:21.529238 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.529134 5850 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-data\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:21.530981 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.530951 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-kube-api-access-lwjs8" pod "9a8a7756-9799-4da8-8afe-39ef1635de72" (UID: "9a8a7756-9799-4da8-8afe-39ef1635de72"). InnerVolumeSpecName "kube-api-access-lwjs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:21.531123 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.531002 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-token" pod "9a8a7756-9799-4da8-8afe-39ef1635de72" (UID: "9a8a7756-9799-4da8-8afe-39ef1635de72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:21.531262 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.531245 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-envoy" pod "9a8a7756-9799-4da8-8afe-39ef1635de72" (UID: "9a8a7756-9799-4da8-8afe-39ef1635de72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:21.531304 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.531290 5850 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-podinfo" pod "9a8a7756-9799-4da8-8afe-39ef1635de72" (UID: "9a8a7756-9799-4da8-8afe-39ef1635de72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:36:21.630762 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.630718 5850 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwjs8\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-kube-api-access-lwjs8\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:21.630762 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.630756 5850 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-envoy\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:21.630990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.630769 5850 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8a7756-9799-4da8-8afe-39ef1635de72-credential-socket\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:21.630990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.630783 5850 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-token\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:21.630990 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.630791 5850 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8a7756-9799-4da8-8afe-39ef1635de72-istio-podinfo\") on node \"ip-10-0-131-244.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:21.875182 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.875151 5850 generic.go:356] "Generic (PLEG): container finished" podID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerID="aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9" exitCode=0 Nov 02 20:36:21.875393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.875202 5850 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" Nov 02 20:36:21.875393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.875241 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" event={"ID":"9a8a7756-9799-4da8-8afe-39ef1635de72","Type":"ContainerDied","Data":"aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9"} Nov 02 20:36:21.875393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.875308 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26" event={"ID":"9a8a7756-9799-4da8-8afe-39ef1635de72","Type":"ContainerDied","Data":"9d55639b0d0899ba6d0c7b64c40bc3d2eaa405f0828acc4f7f62662ef54b3359"} Nov 02 20:36:21.875393 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.875340 5850 scope.go:122] "RemoveContainer" containerID="aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9" Nov 02 20:36:21.884254 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.884234 5850 scope.go:122] "RemoveContainer" containerID="0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35" Nov 02 20:36:21.892453 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.892294 5850 scope.go:122] "RemoveContainer" containerID="aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9" Nov 02 20:36:21.892603 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:36:21.892582 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9\": container with ID starting with aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9 not found: ID does not exist" containerID="aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9" Nov 02 20:36:21.892678 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.892617 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9"} err="failed to get container status \"aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9\": rpc error: code = NotFound desc = could not find container \"aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9\": container with ID starting with aca2673a44a2855d5149bbe458747a4b2c974e100cb18b5fe5d391b996dbeac9 not found: ID does not exist" Nov 02 20:36:21.892678 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.892645 5850 scope.go:122] "RemoveContainer" containerID="0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35" Nov 02 20:36:21.892937 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:36:21.892918 5850 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35\": container with ID starting with 0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35 not found: ID does not exist" containerID="0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35" Nov 02 20:36:21.892991 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.892948 5850 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35"} err="failed to get container status \"0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35\": rpc error: code = NotFound desc = could not find container \"0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35\": container with ID starting with 0d1b044450e126ad79130a68b142a9b8736d5cbec5c760162dd199352e604d35 not found: ID does not exist" Nov 02 20:36:21.985081 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:21.985049 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:22.043680 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:22.043641 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26"] Nov 02 20:36:22.095076 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:22.095042 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5b04055d-openshift-default-7795fd879-pgd26"] Nov 02 20:36:22.148604 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:22.148512 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:22.345912 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:22.345858 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:23.814397 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:23.814355 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" path="/var/lib/kubelet/pods/9a8a7756-9799-4da8-8afe-39ef1635de72/volumes" Nov 02 20:36:25.264549 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:25.264515 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:25.452027 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:25.451985 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:25.782601 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:25.782565 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:27.828621 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:27.828581 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:27.981109 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:27.981073 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:28.134290 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:28.134216 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:32.332130 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:32.332090 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:36.730861 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:36.730825 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:41.123787 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:41.123707 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:44.761242 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:36:44.761202 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:36:45.779171 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:45.779131 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:46.917877 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:46.917844 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:50.169962 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:50.169922 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:54.637747 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:54.637709 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:55.047107 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:55.047018 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j"] Nov 02 20:36:55.098313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:55.098278 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd61f0f8-mxz5j"] Nov 02 20:36:55.813651 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:55.813618 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c58a9c-aed0-4b17-b272-f5c9b6558552" path="/var/lib/kubelet/pods/c1c58a9c-aed0-4b17-b272-f5c9b6558552/volumes" Nov 02 20:36:59.042271 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:36:59.042234 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:03.446875 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:03.446834 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:04.086128 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:04.086093 5850 scope.go:122] "RemoveContainer" containerID="35099c33891a25933405ff4094222be44d2773ae5bdb644259e187aac6b2b7d0" Nov 02 20:37:07.836928 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:07.836888 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:12.222903 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:12.222872 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:14.761170 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:37:14.761128 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:37:16.624586 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:16.624536 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:21.031925 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:21.031886 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:25.424950 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:25.424914 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:26.108304 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:26.108253 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:26.386313 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:26.386279 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:29.829430 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:29.829394 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:34.240239 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:34.240203 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:38.635161 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:38.635103 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:43.036152 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:43.036116 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:44.759986 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:37:44.759953 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:37:47.435525 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:47.435458 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:51.839615 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:51.839578 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:56.238649 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:37:56.238606 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:00.632654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:00.632607 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:04.112983 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:04.112947 5850 scope.go:122] "RemoveContainer" containerID="619bb0ad3257575f9e77aa7c04908e6d8766f5464e885405823bbe830f5fe82e" Nov 02 20:38:05.042164 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:05.042128 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:09.417153 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:09.417068 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:14.041207 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:14.041168 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:14.761545 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:38:14.761505 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:38:18.452301 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:18.452265 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:22.036701 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:22.036662 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:22.186935 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:22.186894 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:26.440084 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:26.440046 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:31.007779 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:31.007740 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:32.409864 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:32.409822 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:32.652428 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:32.652392 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:33.388658 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:33.388619 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:39.203426 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:39.203384 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:39.372809 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:39.372769 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:39.572279 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:39.572187 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:44.761772 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:38:44.761724 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:38:54.472000 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:38:54.471964 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:04.134284 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:04.134248 5850 scope.go:122] "RemoveContainer" containerID="f9a00bb2cdf6e9508ed18da529b0c20d0a6efc70f49813075d21e931c6e8d5f9" Nov 02 20:39:14.761868 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:39:14.761822 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:39:17.046399 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:17.046360 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf"] Nov 02 20:39:17.097660 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:17.097626 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rfcrf"] Nov 02 20:39:17.814035 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:17.814000 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11855b7-41e7-4592-9f61-ccbbdc5b3df8" path="/var/lib/kubelet/pods/e11855b7-41e7-4592-9f61-ccbbdc5b3df8/volumes" Nov 02 20:39:26.851828 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:26.851773 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:37.055249 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:37.055206 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n"] Nov 02 20:39:37.106533 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:37.106493 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45v2n"] Nov 02 20:39:37.813776 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:37.813735 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e91c0ae0-e400-4479-8919-a92261bbd443" path="/var/lib/kubelet/pods/e91c0ae0-e400-4479-8919-a92261bbd443/volumes" Nov 02 20:39:44.761172 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:39:44.761093 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:39:58.893644 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:58.893595 5850 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-w2gts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:39:58.894375 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:39:58.894338 5850 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-w2gts" podUID="3feb82c9-2c7d-46aa-873d-9db55a56ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:40:04.046820 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:04.046750 5850 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh"] Nov 02 20:40:04.098334 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:04.098296 5850 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6476e994-qnrlh"] Nov 02 20:40:05.813649 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:05.813615 5850 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7c5573-c7ee-4876-984c-7f034ee569fd" path="/var/lib/kubelet/pods/ff7c5573-c7ee-4876-984c-7f034ee569fd/volumes" Nov 02 20:40:14.761884 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:40:14.761836 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:40:26.761540 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:26.761506 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6c866c6d6d-md4dq_940a998f-8cbe-42ac-ba71-4b36997feba2/manager/0.log" Nov 02 20:40:27.106927 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:27.106835 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:28.468928 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:28.468890 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:29.990244 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:29.990211 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f6584b448-px4lz_5cea4586-2f7f-46cc-a090-dd11d4b0f7d5/maas-api/0.log" Nov 02 20:40:30.841978 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:30.841946 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-dx6gt_2c0f7860-82ee-4546-b523-7bc62e32d1e4/postgres/0.log" Nov 02 20:40:35.129671 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:35.129623 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rsp86_aed97a99-5750-40b1-be06-80341d05defe/discovery/0.log" Nov 02 20:40:35.669838 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:35.669802 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65657466b-r9pv9_4f99a447-9c59-4e48-866c-1ace82f01793/kube-auth-proxy/0.log" Nov 02 20:40:44.761668 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:40:44.761626 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:40:46.617570 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:46.617537 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-d2chl_6222ddb5-cd09-43db-82f2-01c85b1e63d7/istio-proxy/0.log" Nov 02 20:40:47.340733 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:47.340702 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hszrp_ac5dc2a4-1609-4d43-b359-19adbb7aaeb8/istio-proxy/0.log" Nov 02 20:40:47.553920 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:47.553891 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-k56z5_1d8c479b-a392-4f75-9e43-c25fe3b73ada/istio-proxy/0.log" Nov 02 20:40:47.986930 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:47.986901 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-w2gts_3feb82c9-2c7d-46aa-873d-9db55a56ad68/istio-proxy/0.log" Nov 02 20:40:48.469181 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:48.469150 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m462h_8cc49add-9816-4d83-b8e9-e87fbedd5de2/payload-pre-processing/0.log" Nov 02 20:40:50.491677 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:50.491644 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ks6x5_10bf093b-465e-4039-a1a9-fa51b4262cd7/main/0.log" Nov 02 20:40:50.916287 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:50.916259 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf_6b8c1541-a097-42a4-8094-ce28470d7582/main/0.log" Nov 02 20:40:51.558020 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:40:51.557981 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq_db3b64b1-d8e5-4e64-a5c4-97158401ed5c/main/0.log" Nov 02 20:41:01.441377 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:01.441342 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6c866c6d6d-md4dq_940a998f-8cbe-42ac-ba71-4b36997feba2/manager/0.log" Nov 02 20:41:04.163370 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:04.163331 5850 scope.go:122] "RemoveContainer" containerID="cb2186d06395d04bf79b7d05ae2e3de3914b6815636e5f5e894c5bfcbe759781" Nov 02 20:41:04.172293 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:04.172270 5850 scope.go:122] "RemoveContainer" containerID="a290e8aa411a60d3cb26183a7ad0dc662b05edc7923f245e6783ba09998c7742" Nov 02 20:41:04.181191 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:04.181161 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f6584b448-px4lz_5cea4586-2f7f-46cc-a090-dd11d4b0f7d5/maas-api/0.log" Nov 02 20:41:04.182118 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:04.182099 5850 scope.go:122] "RemoveContainer" containerID="2cdcf2fd1458a8524496ef1d5f39378180334bdf36b79523a8cb0022ef724ee3" Nov 02 20:41:05.042344 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:05.042316 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-dx6gt_2c0f7860-82ee-4546-b523-7bc62e32d1e4/postgres/0.log" Nov 02 20:41:14.761592 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:41:14.761505 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:41:18.790886 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:18.790849 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rsp86_aed97a99-5750-40b1-be06-80341d05defe/discovery/0.log" Nov 02 20:41:19.354134 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:19.354098 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65657466b-r9pv9_4f99a447-9c59-4e48-866c-1ace82f01793/kube-auth-proxy/0.log" Nov 02 20:41:20.327867 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:20.327827 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-d2chl_6222ddb5-cd09-43db-82f2-01c85b1e63d7/istio-proxy/0.log" Nov 02 20:41:21.030174 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:21.030146 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hszrp_ac5dc2a4-1609-4d43-b359-19adbb7aaeb8/istio-proxy/0.log" Nov 02 20:41:21.252150 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:21.252117 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-k56z5_1d8c479b-a392-4f75-9e43-c25fe3b73ada/istio-proxy/0.log" Nov 02 20:41:21.694902 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:21.694867 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-w2gts_3feb82c9-2c7d-46aa-873d-9db55a56ad68/istio-proxy/0.log" Nov 02 20:41:22.165168 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:22.165139 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m462h_8cc49add-9816-4d83-b8e9-e87fbedd5de2/payload-pre-processing/0.log" Nov 02 20:41:23.718177 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:23.718142 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ks6x5_10bf093b-465e-4039-a1a9-fa51b4262cd7/main/0.log" Nov 02 20:41:24.156148 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:24.156115 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b885jqf_6b8c1541-a097-42a4-8094-ce28470d7582/main/0.log" Nov 02 20:41:24.817110 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:24.817076 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ghtzq_db3b64b1-d8e5-4e64-a5c4-97158401ed5c/main/0.log" Nov 02 20:41:27.420985 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:27.420947 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:44.760915 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:41:44.760870 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:41:51.503695 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:41:51.503655 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:14.761239 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:42:14.761194 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:42:22.581360 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:42:22.581325 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:27.755553 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:42:27.755520 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:44.762005 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:42:44.761908 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:42:47.990869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:42:47.990836 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:48.200490 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:42:48.200428 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:59.252411 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:42:59.252376 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r8qkh_13da2219-8d3c-4b28-b6b0-38b51415de22/global-pull-secret-syncer/0.log" Nov 02 20:43:01.028445 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:01.028413 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" Nov 02 20:43:02.356995 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:02.356959 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-244.us-west-2.compute.internal_620b74bb5da064bc73a13b120097bf7c/haproxy/0.log" Nov 02 20:43:08.423525 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.423488 5850 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn"] Nov 02 20:43:08.424039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.423943 5850 cpu_manager.go:414] "containerMap: removing container" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" Nov 02 20:43:08.424039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.423955 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" Nov 02 20:43:08.424039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.423968 5850 cpu_manager.go:414] "containerMap: removing container" podUID="35a435e7-b92e-4e5f-bc00-4aa007e47cfd" containerName="maas-api" Nov 02 20:43:08.424039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.423973 5850 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a435e7-b92e-4e5f-bc00-4aa007e47cfd" containerName="maas-api" Nov 02 20:43:08.424180 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.424040 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a435e7-b92e-4e5f-bc00-4aa007e47cfd" containerName="maas-api" Nov 02 20:43:08.424180 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.424053 5850 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8a7756-9799-4da8-8afe-39ef1635de72" containerName="istio-proxy" Nov 02 20:43:08.427307 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.427286 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.481302 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.481258 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjlh5\"/\"openshift-service-ca.crt\"" Nov 02 20:43:08.523959 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.523923 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-sys\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.523959 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.523962 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6qn\" (UniqueName: \"kubernetes.io/projected/c1017178-da86-441f-a902-b333dad9527e-kube-api-access-sx6qn\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.524189 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.523982 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-proc\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.524230 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.524180 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-podres\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.524278 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.524246 5850 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-lib-modules\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.533038 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.533006 5850 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pjlh5\"/\"default-dockercfg-2k858\"" Nov 02 20:43:08.533293 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.533259 5850 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjlh5\"/\"kube-root-ca.crt\"" Nov 02 20:43:08.536178 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.536142 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn"] Nov 02 20:43:08.625549 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.625509 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-podres\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.625750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.625571 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-lib-modules\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.625750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.625610 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-sys\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.625750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.625632 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6qn\" (UniqueName: \"kubernetes.io/projected/c1017178-da86-441f-a902-b333dad9527e-kube-api-access-sx6qn\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.625750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.625650 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-podres\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.625750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.625660 5850 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-proc\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.625750 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.625720 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-lib-modules\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.625989 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.625750 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-proc\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.625989 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.625763 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1017178-da86-441f-a902-b333dad9527e-sys\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.684110 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.684030 5850 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6qn\" (UniqueName: \"kubernetes.io/projected/c1017178-da86-441f-a902-b333dad9527e-kube-api-access-sx6qn\") pod \"perf-node-gather-daemonset-znjqn\" (UID: \"c1017178-da86-441f-a902-b333dad9527e\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:08.739601 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:08.739562 5850 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:09.108292 ip-10-0-131-244 kubenswrapper[5850]: W1102 20:43:09.108255 5850 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc1017178_da86_441f_a902_b333dad9527e.slice/crio-013b416a42a3512dbe572f3c7880b8f1397e1e350804c7476e21b892b3150a04 WatchSource:0}: Error finding container 013b416a42a3512dbe572f3c7880b8f1397e1e350804c7476e21b892b3150a04: Status 404 returned error can't find the container with id 013b416a42a3512dbe572f3c7880b8f1397e1e350804c7476e21b892b3150a04 Nov 02 20:43:09.109621 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:09.109592 5850 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn"] Nov 02 20:43:10.103283 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:10.103240 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" event={"ID":"c1017178-da86-441f-a902-b333dad9527e","Type":"ContainerStarted","Data":"f4584ebb186b6527e5df36e67073911d5902de61ab1bd3c701dd4bd7e58914f4"} Nov 02 20:43:10.103713 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:10.103295 5850 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" event={"ID":"c1017178-da86-441f-a902-b333dad9527e","Type":"ContainerStarted","Data":"013b416a42a3512dbe572f3c7880b8f1397e1e350804c7476e21b892b3150a04"} Nov 02 20:43:10.103713 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:10.103323 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:10.229905 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:10.229831 5850 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" podStartSLOduration=2.229810255 podStartE2EDuration="2.229810255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:10.218798282 +0000 UTC m=+4448.561187364" watchObservedRunningTime="2026-11-02 20:43:10.229810255 +0000 UTC m=+4448.572199336" Nov 02 20:43:12.097016 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:12.096979 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfba8c33-8580-4b90-ae51-b31988a27692/alertmanager/0.log" Nov 02 20:43:12.483627 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:12.483594 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfba8c33-8580-4b90-ae51-b31988a27692/config-reloader/0.log" Nov 02 20:43:12.870726 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:12.870698 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfba8c33-8580-4b90-ae51-b31988a27692/kube-rbac-proxy-web/0.log" Nov 02 20:43:13.256640 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:13.256548 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfba8c33-8580-4b90-ae51-b31988a27692/kube-rbac-proxy/0.log" Nov 02 20:43:13.647879 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:13.647849 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfba8c33-8580-4b90-ae51-b31988a27692/kube-rbac-proxy-metric/0.log" Nov 02 20:43:14.035436 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:14.035362 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfba8c33-8580-4b90-ae51-b31988a27692/prom-label-proxy/0.log" Nov 02 20:43:14.422341 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:14.422308 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cfba8c33-8580-4b90-ae51-b31988a27692/init-config-reloader/0.log" Nov 02 20:43:14.760968 ip-10-0-131-244 kubenswrapper[5850]: E1102 20:43:14.760879 5850 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vngml" containerName="extractor" Nov 02 20:43:15.207767 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:15.207731 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-frwtw_43a8f63a-57a0-4be5-b55f-621b67342950/kube-state-metrics/0.log" Nov 02 20:43:15.628564 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:15.628528 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-frwtw_43a8f63a-57a0-4be5-b55f-621b67342950/kube-rbac-proxy-main/0.log" Nov 02 20:43:16.016662 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:16.016564 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-frwtw_43a8f63a-57a0-4be5-b55f-621b67342950/kube-rbac-proxy-self/0.log" Nov 02 20:43:16.119853 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:16.119826 5850 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-znjqn" Nov 02 20:43:16.407229 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:16.407204 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-868bfbd57c-9kws8_48eccb46-4917-49bb-b8f0-f40cf28de5f6/metrics-server/0.log" Nov 02 20:43:16.796524 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:16.796436 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-2hv87_107e58b3-1281-416a-b713-8efc20576e9f/monitoring-plugin/0.log" Nov 02 20:43:17.188701 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:17.188669 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cswh5_daf0d624-3b0c-4ceb-897a-c9d8b54213e2/node-exporter/0.log" Nov 02 20:43:17.574200 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:17.574120 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cswh5_daf0d624-3b0c-4ceb-897a-c9d8b54213e2/kube-rbac-proxy/0.log" Nov 02 20:43:17.960492 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:17.960444 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cswh5_daf0d624-3b0c-4ceb-897a-c9d8b54213e2/init-textfile/0.log" Nov 02 20:43:24.594770 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:24.594735 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4rpfq_2867a688-7f22-4419-a69c-aef174888edc/prometheus-operator/0.log" Nov 02 20:43:24.979767 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:24.979735 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4rpfq_2867a688-7f22-4419-a69c-aef174888edc/kube-rbac-proxy/0.log" Nov 02 20:43:25.372039 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:25.372010 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-mkf7t_a2de0d2d-3838-4979-b2ec-c3ed7df1beb9/prometheus-operator-admission-webhook/0.log" Nov 02 20:43:25.763634 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:25.763560 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57ddbd48bb-l5d62_e0180ace-54a9-4796-a2ab-f069f965573c/telemeter-client/0.log" Nov 02 20:43:26.153640 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:26.153612 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57ddbd48bb-l5d62_e0180ace-54a9-4796-a2ab-f069f965573c/reload/0.log" Nov 02 20:43:26.542819 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:26.542744 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57ddbd48bb-l5d62_e0180ace-54a9-4796-a2ab-f069f965573c/kube-rbac-proxy/0.log" Nov 02 20:43:26.936092 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:26.936057 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c99bbd67-6qj9p_de1657c9-1cb9-44bb-b505-adf726ab2540/thanos-query/0.log" Nov 02 20:43:27.324355 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:27.324263 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c99bbd67-6qj9p_de1657c9-1cb9-44bb-b505-adf726ab2540/kube-rbac-proxy-web/0.log" Nov 02 20:43:27.710194 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:27.710146 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c99bbd67-6qj9p_de1657c9-1cb9-44bb-b505-adf726ab2540/kube-rbac-proxy/0.log" Nov 02 20:43:28.096000 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:28.095924 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c99bbd67-6qj9p_de1657c9-1cb9-44bb-b505-adf726ab2540/prom-label-proxy/0.log" Nov 02 20:43:28.103516 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:28.103461 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:28.333869 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:28.333831 5850 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wwnx_d71e26dd-fe22-44a6-9b04-bf30e4aef71a/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:28.481654 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:28.481628 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c99bbd67-6qj9p_de1657c9-1cb9-44bb-b505-adf726ab2540/kube-rbac-proxy-rules/0.log" Nov 02 20:43:28.867755 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:28.867723 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64c99bbd67-6qj9p_de1657c9-1cb9-44bb-b505-adf726ab2540/kube-rbac-proxy-metrics/0.log" Nov 02 20:43:31.100051 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:31.100017 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rsp86_aed97a99-5750-40b1-be06-80341d05defe/discovery/0.log" Nov 02 20:43:31.606604 ip-10-0-131-244 kubenswrapper[5850]: I1102 20:43:31.606570 5850 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65657466b-r9pv9_4f99a447-9c59-4e48-866c-1ace82f01793/kube-auth-proxy/0.log"